Evolution of a Vertebrate Social Decision-Making Network

See allHide authors and affiliations

Science  01 Jun 2012:
Vol. 336, Issue 6085, pp. 1154-1157
DOI: 10.1126/science.1218889


Animals evaluate and respond to their social environment with adaptive decisions. Revealing the neural mechanisms of such decisions is a major goal in biology. We analyzed expression profiles for 10 neurochemical genes across 12 brain regions important for decision-making in 88 species representing five vertebrate lineages. We found that behaviorally relevant brain regions are remarkably conserved over 450 million years of evolution. We also find evidence that different brain regions have experienced different selection pressures, because spatial distribution of neuroendocrine ligands are more flexible than their receptors across vertebrates. Our analysis suggests that the diversity of social behavior in vertebrates can be explained, in part, by variations on a theme of conserved neural and gene expression networks.

Animals have evolved flexible strategies that allow them to respond to their social environment by integrating the salience of an external stimulus with internal physiological cues into an adaptive behavioral response (1, 2). Although individual fitness depends on displaying adaptive behavior patterns (e.g., reproductive or aggressive behavior) in a context-appropriate manner, it has been difficult to examine the evolution of underlying neural and molecular mechanisms because of diversity in ecology, alternative behavior tactics, and brain structure across vertebrates (3, 4). At the same time, this variation provides a unique opportunity to determine the extent to which variance in neural gene expression underlies behavioral adaptations within and across species. To better understand the large-scale patterns of neurochemical evolution across vertebrates, we analyzed the expression profiles for 10 gene products across 12 brain regions that encode and process environmental and physiological cues important for decision-making in 88 species representing five major vertebrate lineages.

Our analysis of the neurochemical evolution of the vertebrate brain focuses on two neural circuits originally described in mammals and since expanded to other vertebrate lineages (5): (i) the social behavior network, which in concert with sex steroid and neuropeptide hormones regulates social behavior such as reproduction, aggression, and parental care (6, 7), and (ii) the mesolimbic reward system, which is generally assumed to evaluate stimulus salience via dopaminergic signaling (8, 9). Although the evolutionary antecedents of these neural circuits (especially regions of the basal ganglia and limbic system) have been debated for the past century (10), it is currently recognized that they were already present in the common ancestor of sarcopterygians (including tetrapods) and actinopterygians (ray-finned fishes) (5, 7, 11). These two circuits constitute a larger integrated social decision-making (SDM) network that governs stimulus evaluation and behavior across vertebrates (1, 5). Within this framework, we chose to examine the distribution of behaviorally relevant gene products associated with the dopaminergic system [tyrosine hydroxylase (TH), dopamine D1 receptor, or DARPP-32], sex steroid hormone signaling (aromatase and nuclear receptors for estrogen, androgen, and progesterone), and nonapeptide systems (vasopressin, oxytocin, and their receptors). These pathways play fundamental roles in regulating complex social behavior in all vertebrate taxa (12, 13); however, we excluded several neurochemical systems of obvious importance, such as other aminergic, opioid, and neuropeptide pathways, as well as the D2-like dopamine receptor, because of a lack of comprehensive information available across taxa.

To compare the neurochemical profiles of brain regions across vertebrates, we analyzed decades of research in vertebrate neurochemistry by examining published neuroanatomical micrographs presenting mRNA or protein expression. Within each species, we first ascertained whether a gene product of interest was present or absent in SDM network nodes (Fig. 1) and then determined the consensus for that vertebrate lineage. Some brain regions given mammalian names in our analysis here may not represent discrete homologs of mammalian structures (5). Additionally, we omitted the prefrontal cortex from our comparative analysis, even though it is considered part of the reward system in mammals, because its evolutionary antecedent in nonmammalian vertebrates is unclear (14). Furthermore, we combined distribution data for paralogous genes within a species (e.g., estrogen receptors ERα and ERβ in vertebrates, teleost-specific androgen receptor duplicates ARα and ARβ).

Fig. 1

Neural basis of behavioral diversity. The social behavior network (yellow) and mesolimbic reward system (blue) are two important neural networks regulating behavior in vertebrates and have functional connections (green) between the circuits. The mammalian nomenclature used in nonmammalian lineages does not necessarily imply discrete (one-to-one) homologs of mammalian structures. [Adapted from (5)] AH, anterior hypothalamus; BNST/meAMY, bed nucleus of the stria terminalis/medial amygdala; HIP, hippocampus; LS, lateral septum; NAcc, nucleus accumbens; PAG, periaqueductal gray/central gray; VMH, ventromedial hypothalamus; VP, ventral pallidum; VTA, ventral tegmental area.

When comparing the distribution of neurochemicals across vertebrates, we observed several notable characteristics. First, the neurochemical profiles of the SDM network have been remarkably conserved throughout vertebrate evolution (Fig. 2), consistent with their important roles in regulating behavior in all vertebrates. Secondly, an unsupervised clustering of neurochemical distributions across vertebrates segregated receptor distributions from ligands. When compared with receptors (Fig. 2 top), the spatial distribution of ligands (bottom) is significantly more restricted (χ2 = 9.00; P = 0.011). This analysis does not include aromatase, an enzyme that produces a ligand by converting testosterone to estrogen yet behaves like a receptor in terms of its wide distribution. Lastly, there appears to be more variation in where the ligands are produced (i.e., distribution of dopaminergic and neuropeptide-producing cells) than variation in where receptors are located, suggesting that the spatial distributions of neuroendocrine ligands are evolutionarily more flexible than those of receptors.

Fig. 2

Analysis of vertebrate patterns in brain neurochemistry. Overall patterns in neurochemistry in the social decision-making network are shown across vertebrates. Data for each vertebrate lineage are represented in the columns, and each row represents one gene in one brain region that is either present (orange) or absent (purple). White indicates that no data are available (n/a). The number of species in each vertebrate lineage represented in the analysis is shown at the bottom of each column. The dendrogram (top) is the result of unsupervised hierarchical clustering of the vertebrate neurochemical patterns.

To explore the flexiblility of neurochemical gene expression patterns across evolutionary time in a more quantitative manner, we calculated a divergence score (D) for each neurochemical gene product and brain region. We mapped distributions for each gene product across all SDM network nodes onto a vertebrate phylogeny and calculated D as the number of changes observed in the parsimonious model, where D > 0 indicates that one or more changes have occurred since tetrapods and ray-finned fishes shared their last common ancestor about 450 million years ago (Ma) (15). Figure 3A shows divergence scores for each brain region (columns) and neurochemical gene product (rows). The most-conserved (Davg = 0.0) brain regions in our analysis are the basolateral amygdala (blAMY) and the preoptic area (POA), whereas the striatum (Str) is the least conserved (Davg = 0.7). The average divergence scores for each brain region within either the reward system or social behavior network did not indicate that one system had undergone more changes than the other during vertebrate evolution (Fig. 3B; Mann-Whitney U14 = 27.5, P = 0.662). Furthermore, changes in neurochemical profiles in these two systems are uniformly distributed across the major transitions of vertebrate evolution (Fig. 3, C and D).

Fig. 3

Conservation of neural systems regulating behavior. (A) Divergence scores for each neurochemical gene (rows) in each brain region (columns) were calculated according to a parsimonious model on the vertebrate phylogeny (top right). Red indicates a score of zero (most conserved), yellow indicates a score of one, and purple indicates a score of two (less conserved). Divergence scores were compared between brain networks (B) or gene class (E) and are shown as box and whisker plots. Asterisk (*) indicates significance with Mann-Whitney U test, P = 0.038. Divergence score at each node was plotted across evolutionary time for each brain network (C and D) and for each gene class (F and G). Abbreviations as in Fig. 1. ARO, aromatase.

The neurochemical gene products whose distribution varies the most across the SDM network are TH (Davg = 0.6), arginine vasopressin (AVP; Davg = 0.5), and oxytocin (OXY; Davg = 0.4), whereas the profiles of the oxytocin receptor (OTR; Davg = 0.0) and the progesterone receptor (PR; Davg = 0.0) are the most conserved. Profiles of both the androgen receptor (AR; Davg = 0.2), and the dopamine D1 receptor (D1aR; Davg = 0.3) exhibit little variation. To further examine potential differences in the spatial distribution of ligands and their receptors across vertebrate lineages, we averaged the divergence scores for each neurochemical gene product across brain regions and found that the variation in the spatial distribution of ligands (TH, aromatase, AVP, and OXY) is significantly greater than the variation in spatial distribution of receptors [AR, vasopressin 1a receptor (V1aR), D1aR, ER, OTR, and PR; Fig. 3E] (Mann-Whitney U10 = 2, P = 0.038). Furthermore, the distribution of ligands exhibits a very heterogeneous temporal pattern with two dramatic changes (Fig. 3F). First, the number of SDM network nodes that contain TH-producing cells changed dramatically 450 Ma when the lineages that gave rise to teleosts (actinopterygians) and tetrapods (sarcopterygians) shared their last common ancestor, although without an outgroup we cannot determine whether TH expression was lost or gained in these brain areas. Second, several SDM network nodes gained neuropeptide (AVP and OXY)–expressing cells when birds diverged from reptiles ~220 Ma or even longer ago. The information on reptiles in our analysis is largely limited to lepidosaurians (lizards and snakes), because data on turtles and crocodilians are sorely lacking. The latter group would be particularly interesting because of its close relationship with birds in the archosaurian clade and because of the complex social behavior displayed by many of its representatives. Conversely, very few changes occurred in the spatial distribution of receptors over the past 450 million years (Fig. 3G). Aromatase could reasonably be grouped with either ligands or receptors, although this does not influence our results. To be conservative, we have grouped aromatase in the ligand class.

Because brain regions subserve different behaviorally relevant functions (5), we hypothesized that different SDM network nodes might have been subjected to varying selection pressures regarding their neurochemical profiles. We analyzed the neurochemical pattern of each brain region separately and found that brain regions show different degrees of neurochemical conservation across vertebrates (Fig. 4). We discuss here only the most-conserved and most-diverged brain regions (results for remaining regions are in fig. S1). The neurochemical profile of the preoptic area, a neuroendocrine relay station (16), is completely conserved, likely because of its fundamental role in coordinating and regulating basic behavioral and physiological functions. In contrast, neurochemical gene expression is most variable in the striatum, a multimodal receptive structure involved in reward processing and voluntary motor control (9). The extent to which the striatum receives pallial versus thalamic projections increased considerably at the anamniote-amniote and nonmammalian-mammalian transitions (17), a pattern that likely resulted in major functional shifts and resembles the changes in striatal gene profiles we have observed (Fig. 3C).

Fig. 4

The extent of conservation varies across brain regions. Patterns in neurochemistry are shown for the preoptic area (top) and striatum (bottom) where genes (rows) are either present (orange), absent (purple), or unknown (n/a, white) within each vertebrate lineage (columns). All other brain regions are shown in fig. S1.

We next asked whether the variation seen across vertebrates might be confounded by species-level variation within a lineage. We examined the spatial patterns of dopamine-producing and AVP cells at the species level, because they are the best studied and most divergent across taxa. We found surprisingly little presence/absence variation within a vertebrate lineage in the distribution of dopaminergic or AVP-producing cells (fig. S2), suggesting that the patterns observed across vertebrates are not explained by variation within each lineage. However, within teleosts sites of dopamine production appear to show greater variation, possibly as the results of a more ancient origin and/or additional genome duplication events experienced by this group (18). Differences between closely related species that differ in social behavior have generally been reported as quantitative rather than qualitative (i.e., total presence or absence) with the exception of nonapeptide receptor expression in some mammalian and avian species (19, 20). Our analysis suggests that variation in neurochemical profiles within or across species from the same lineage is generally quantitative, as has been described within species with alternative phenotypes (21), between sexes (22), between species (23), and also in invertebrate behavioral systems (24), whereas large-scale patterns in gene expression variation across vertebrate lineages are generally qualitative (i.e., presence or absence).

We have examined the neurochemical profiles of brain regions important in social decision-making and found that, whereas the SDM network has overall been remarkably conserved over 450 million years of evolution, the sites of ligand production are evolutionarily more flexible than where their receptors are expressed, likely the consequence of two major expansions in ligand production sites ~450 and 222 Ma (2527). The high conservation of receptor distributions may be explained by the ancient and conserved responses by animals to challenges and opportunities in their environment. Conversely, lineage differences in life history and ecology, coupled with phylogenetic constraints, might determine where and how social signals of different modalities are weighted and processed in the brain, possibly aided by differences in the local production of ligands. As a consequence, shifts in ligand expression to new sites in the brain resulting from small developmental changes (28) might accompany the evolution of novel life history strategies or social systems, consistent with the known pleiotropic roles of many of these ligands. Recent investigations into the evolution of developmental mechanisms have similarly observed that the appearance of novel body plans or color patterns is often associated with changes in the spatial expression patterns of morphogenetically important ligands (29). On the other hand, receptors are generally associated with complex intracellular machinery and thus may be more restricted in spatial distribution changes than ligands, given that such a shift must also require altering the spatial distribution of many other proteins. Our analysis also suggests that macro- and microevolutionary processes might be dissociated, given that behavioral variation within a population or across closely related species can arise because of quantitative differences in receptor or ligand distributions (19, 24), although more comparative studies of these and other neurochemicals in nonmammalian taxa are needed to better understand how the social brain evolved with changes in brain gene expression.

Because the brain regions that govern social behavior function within a network, the integrative and systems-level approach we have used here opens up new avenues toward understanding the neural and molecular architecture of social behavior and its evolution. We suggest that the diversity of social behavior in vertebrates can be explained at least in part by variations on a conserved theme of neural and gene expression networks. Thus, our analysis provides a framework that will greatly facilitate the search for molecular universals underlying social behavior (30).

Supplementary Materials

Materials and Methods

Figs. S1 and S2

Tables S1 to S10

References (31180)

References and Notes

  1. Acknowledgments: The data reported in this paper are tabulated in the supplementary materials. We thank M. Cummings, S. Phelps, A. Pollen, M. Ryan, and R. Wong for helpful comments on earlier versions of this manuscript; members of the Hofmann laboratory, the Center for Brain, Behavior, and Evolution at UT Austin, and the Sociogenomics Initiative, especially G. Robinson, for discussions; and D. Canatella, E. Martins, and T. Streelman for advice on the evolutionary analysis. The work was supported by NSF Doctoral Dissertation Improvement Grant 1011253 to L.A.O. and NSF grant 0843712, the Alfred P. Sloan Foundation, a Dwight W. and Blanche Faye Reeder Centennial Fellowship in Systematic and Evolutionary Biology, and an Institute for Cellular and Molecular Biology Fellowship to H.A.H. The authors declare no competing financial interests. L.A.O. gathered all data for analysis; L.A.O. and H.A.H. conceived of the project, analyzed data, and wrote the paper.
View Abstract

Stay Connected to Science

Navigate This Article