Homology, neocortex, and the evolution of developmental mechanisms

See allHide authors and affiliations

Science  12 Oct 2018:
Vol. 362, Issue 6411, pp. 190-193
DOI: 10.1126/science.aau3711


The six-layered neocortex of the mammalian pallium has no clear homolog in birds or non-avian reptiles. Recent research indicates that although these extant amniotes possess a variety of divergent and nonhomologous pallial structures, they share a conserved set of neuronal cell types and circuitries. These findings suggest a principle of brain evolution: that natural selection preferentially preserves the integrity of information-processing pathways, whereas other levels of biological organization, such as the three-dimensional architectures of neuronal assemblies, are less constrained. We review the similarities of pallial neuronal cell types in amniotes, delineate candidate gene regulatory networks for their cellular identities, and propose a model of developmental evolution for the divergence of amniote pallial structures.

The six-layered neocortex is shared by all extant mammals but is absent from nonmammals, including birds and non-avian reptiles (1). The question of where it came from has been disputed for more than 150 years (2). Neuroanatomists in the early 20th century proposed that the neocortex represents a phylogenetic addition to a primitive reptilian brain and that it was responsible for those cognitive abilities thought to be unique to mammals (2). This scenario implies the rapid evolutionary genesis of the neocortex, seemingly from nothing, while leaving open the question of how nonmammals operate without a neocortex equivalent. However, de novo addition is not a mechanistic explanation for the origin of evolutionary novelty. To understand neocortex origins, we must search for homologous neocortical features shared by mammals and their relatives.


Biological structures are organized and evolve at multiple levels: cell types, neural circuits, molecules, and embryological territories (35). These levels can evolve independently of one another, and homology at one level does not require conservation at others. Although the neocortex is present only in mammals, some of its components may evolutionarily predate the common ancestor that mammals share with birds and reptiles. If true, the evolutionary origin of the neocortex could be explained, at least in part, by reorganization of old components at other hierarchical levels. Such a mechanism would resolve the apparent conflict between morphological novelty and the historical continuity required of evolving biological systems.

A current evolutionary developmental perspective focuses on cell types for considerations of homology (6, 7). The claims of this view are that (i) cell types are elemental units of animal organization, (ii) cell identity is specified during development by evolutionarily stable collections of transcription factors, (iii) homologous cell types can exist in distantly related animals, and (iv) cell types can acquire species-specific states through evolutionary modifications that do not compromise their identities. Here, we review recent and classical findings indicating that core neuronal cell types of neocortical circuits are shared across mammals, reptiles, and birds. Lineage-specific developmental mechanisms may differentially arrange these conserved cell types into a wide variety of derived structural architectures, modify their morphologies and developmental origins, and add novel cell types.

Homologous cell types and circuitry in the amniote pallium

The neuroanatomical structures in the dorsal telencephalon, or pallium, of amniotes (mammals, birds, and non-avian reptiles) look very little alike (Fig. 1). Neuronal cell types of the mammalian neocortex are organized into layers. In contrast, the pallium of birds and other reptiles (the sauropsids) features a vast territory, the dorsal ventricular ridge (DVR), of neuronal cell body clusters called nuclei. In non-avian reptiles, a three-layered dorsal cortex lies above the DVR. Birds likely descended from reptiles with a dorsal cortex; if so, this cortex became modified into a second nuclear complex known as the Wulst (8). These anatomical disparities have confounded efforts to establish homologies based on morphology alone—an approach that has generated incompatible interpretations (2, 4, 5). Comparisons made at the level of neuronal cell types have yielded a more coherent story.

Fig. 1 The amniotes and their pallial anatomies.

Amniotes include mammals and sauropsids (birds and non-avian reptiles). Tuataras, together with snakes and lizards (not shown), form the lepidosaurs. A second major group of sauropsids, the archosaurs, includes birds and crocodilians. Schematic tracings of telencephalon anatomies are shown as left-side coronal cross sections with medial to the right and dorsal at the top. All amniotes have a ventral telencephalon (VT, gray shading) and a dorsal telencephalon, or pallium (peach shading). The mammalian pallium includes the neocortex (Ncx), piriform cortex (CPi), hippocampus (not shown), and amygdala (not shown). The pallium in non-avian reptiles includes a dorsal ventricular ridge (DVR) and a cerebral cortex with medial (MC), dorsal (DC), and lateral (LC) divisions. The bird DVR contains the ventral part of the mesopallium (M), the nidopallium (N), the entopallium (E), and the arcopallium (not shown). Birds have a medial hippocampus (Hp), a lateral piriform cortex (CPi), and a dorsally located Wulst that includes the dorsal mesopallium and the hyperpallium (H). Drawings are not to scale.


The avian pallium does not contain cortical layers, but its excitatory neurons form input and output connections that are very similar to neocortical circuitry. In both the mammalian neocortex and the avian pallium, distinct populations of neurons either receive direct sensory input from the thalamus or convey motor output through projections to the brainstem. One hypothesis proposes that the connections are similar because the pallial input and output neurons were inherited from a common ancestor (Fig. 2, green input cells and red output cells) (9). Comparative molecular studies confirmed key aspects of this cell type homology hypothesis. The selective expression of conserved neocortical layer markers distinguishes pallial input and output neurons in a wide range of amniotes, rendering convergent evolution an unparsimonious explanation for the connectional similarities (1012).

Fig. 2 Evolution of excitatory neocortical cell types and circuitry.

The common ancestor of amniotes is hypothesized to have had input (green), output (red), and intratelencephalic (IT, blue) pallial neurons. Input neurons receive primary sensory information from the dorsal thalamus (dTh), whereas output neurons extend axons from the pallium to the brainstem (Bst). IT neurons serve as a relay between input and output neurons and additionally project to the striatum. These three principal pallial cell types were reorganized into the divergent architectures of the neocortex, dorsal cortex, Wulst, and DVR (not shown). Candidate transcriptional regulators of cell identity are listed beside the ancestral cell types (10, 1618). The evolutionary origin of neocortical corticothalamic neurons (black cells in L6) is not known (18). The pallial inhibitory interneurons, derived from the ventral telencephalon, are not indicated [but see (16)]. HA, hyperpallium apicale; IHA, interstitial nucleus of the hyperpallium apicale; L, layer; Md, dorsal mesopallium.


The lesson from the cell type homology hypothesis, and from its later molecular affirmation, is that natural selection preserves the core cell types and circuits of the amniote pallium, with less constraint on the morphology of pallial structures. Pallial input and output neurons in amniotes have maintained their defining connectional and molecular characteristics but assemble into taxon-specific architectures, forming stacked cortical layers in the neocortex, nuclei in the DVR, and, in the turtle cortex, spatially segregated cortical fields (10).

Intratelencephalic neurons and the evolution of higher cognitive abilities

Behavioral studies established the cognitive prowess of birds, some of which possess impressive skills in problem solving, memory, and tool use. The evolution of avian cognitive abilities is attributed to their massive, cell-dense pallium and to two large pallial territories, the mesopallium and the nidopallium (Fig. 1) (13). These territories serve as integrative centers and can be viewed as functionally analogous either to neocortical upper-layer neurons or to neocortical association areas such as the prefrontal cortex (14). The mesopallium and nidopallium receive axons from the pallial primary sensory input nuclei, interconnect with one another, and project in turn to the pallial motor output nuclei (15). They do not structurally resemble any district of the mammalian pallium, and their relationships even to non-avian reptile structures were until recently obscure (1).

Insight into the evolution of the mesopallium has been gained through molecular studies in multiple bird and non-avian reptile species. RNA sequencing in chickens and single-cell transcriptomics in turtles and lizards identified multiple transcription factors as diagnostic of conserved pallial cell populations (16, 17). Six transcription factors characterize the avian mesopallium (17), which, as defined by Jarvis et al. (11), includes a dorsal division associated with the Wulst and a ventral division in the DVR. Gene expression experiments in alligators revealed a similar bipartite organization, with mesopallium-like cells in the dorsal cortex and a mesopallium in the DVR (17, 18). Cell populations in turtle and lizard cortices also express several of the mesopallium transcription factors, indicating that mesopallium-like cell types are shared across sauropsids (16).

The mesopallium transcription factors are expressed in the mammalian neocortex, but not by any specific layer or area. Rather, they identify neocortical intratelencephalic (IT) neurons, found in both upper and deep neocortical layers in all neocortical areas (17, 19). Moreover, two mesopallium transcription factors, Satb2 and Bcl11a, are genetically required in mice to specify an IT connectional phenotype (2022), which suggests that the mesopallium transcription factors may form a cell type gene regulatory network (Fig. 2, blue IT cells). Thus, the IT neurons, like those in the avian mesopallium, are a class of excitatory neurons linking the input with output populations and having axons that remain within the telencephalon (15, 19).

Connectional and molecular data together support a model that pallial IT neurons, along with input and output neurons, were present in the amniote last common ancestor (LCA) where they formed a conserved circuit motif (Fig. 2). We propose that a limited ancestral stock of pallial IT neurons independently diversified to give rise to neurons in the reptile cortex, the DVR, the Wulst, and the IT neurons of the neocortex. The evolutionary expansion of pallial IT neurons may be a common neuroanatomical substrate for the evolution of higher cognitive abilities in mammals and birds (17). That is, brainy amniotes such as great apes and crows have disproportionately expanded pallial IT cell populations, and this extra integrative circuitry may have facilitated the evolution of complex cognition (14).

Specification and evolution of pallial neuronal cell types

Cell type identity is established during development by networks of transcriptional regulators that are relatively stable across phylogeny (7). Comparative studies suggest that pallial neuronal cell types may develop and evolve by the same general principles as other metazoan cell types and, in particular, they reveal a striking evolutionary correlation between transcription factor expression and cell type defined by connectivity (Fig. 2). For example, the paralogous genes RORA and RORB, both of which regulate the differentiation of neocortical input neurons (23, 24), are expressed by pallial input cells of birds and reptiles (10, 11, 16, 18). These two transcription factors, together with SATB1 (16, 18), may participate in a conserved gene regulatory network for the identity of pallial input cells, including their connections and physiological properties (Fig. 2, ancestral input cell). This possibility demands further study in bird and reptile embryos where in ovo gene manipulation experiments are straightforward (17, 2527).

Whereas the transcription factor profiles of input and IT cell types are similar across amniotes, other features of pallial neurons are clearly divergent. Many excitatory neurons of the mammalian neocortex are pyramidal in morphology, with a layer-spanning apical dendrite, whereas most avian pallial neurons are stellate. The developmental mechanisms regulating taxon-specific neuronal morphologies are not known, nor are the consequences of these divergent morphologies for information processing (10, 28). Substantial group differences exist also at the molecular level, as some sauropsid pallial neurons express transcription factors in combinations not observed in the neocortex (16). Moreover, between sauropsids and mammals, there remain pallial cell types for which molecular and connectional data do not suggest clear homologies (Figs. 2 and 3) (17). Reconfiguration of transcription factor expression patterns, protein interactions, and transcriptional targets may be a means by which neuronal cell types evolve and diversify (6, 7, 16).

Fig. 3 Anatomical transformations during the evolutionary diversification of amniote pallial structures.

The hypothetical amniote last common ancestor had an architecturally simple pallium, which contained input (green), output (red), IT (blue), and hippocampal (orange) cell types. The ancestral pallium underwent independent evolutionary transformations to give rise to the mammalian neocortex and the sauropsid DVR, dorsal cortex, and Wulst. The brown arrows denote an expansion of the lateral pallium with the origin of the DVR in early sauropsids. This expansion may have accompanied the origin of non-input nidopallium cells (brown). DVR input nuclei are embedded within the nidopallium. v, ventricle; VT, ventral telencephalon.


A model for the developmental evolution of amniote pallial structures

There is no fossil record for the internal organization of vertebrate brains. To propose mechanisms of anatomical transformation, we can only construct models of the ancestral anatomy by comparing extant forms. Evidence reviewed here suggests that the amniote LCA had input, output, and IT cells (Fig. 3, green, red, and blue cells, respectively) (17). Moreover, recent work establishes that hippocampal dentate gyrus– and CA field–like neurons are also ancestral (Fig. 3, orange cells) (16, 18). Our model of anatomical transformations incorporates these findings and makes two assumptions: (i) The pallial anatomy of the amniote LCA was simpler than in extant forms, and (ii) the spatial distribution of cell types resembled that of extant sauropsids more than it resembled the neocortical organization.

The amniote LCA

Pallial morphology in amphibians, the closest living outgroup to amniotes, is simple (29). Neurons are typically distributed next to the ventricle with few structural specializations. There is no DVR. We propose that the amniote LCA possessed an architecturally simple pallium, similar to that of extant amphibians, but one with extensive primary sensory inputs and local integrative circuitry mediated by IT neurons (Fig. 3, Amniote LCA).

All extant amniotes, including mammals, receive somatosensory and visual information in the dorsal pallium and visual and auditory information in the lateral pallium (30, 31). The model amniote LCA therefore has two populations of input cells: a dorsal group that gave rise to input cells in the sauropsid dorsal cortex and Wulst, and a lateral group that gave rise to input populations in the DVR. We speculate that a population of IT cells was intercalated between the dorsal and lateral input groups, much as an IT cell population separates dorsal and lateral input cell groups in birds and alligators (17, 18). Output cells have been described in the avian Wulst, the avian DVR, and the reptile dorsal cortex, so we suggest that output cells were intermingled with the input cells in the amniote LCA.

The sauropsid LCA

The first developmental transformation would be the acquisition of the DVR in the sauropsid lineage. At this transition, the targets of the dorsal and lateral ascending sensory pathways became individualized into morphologically distinct territories: a dorsal cortex-like structure and a lateral DVR, each with different modes of development. This dorsal and lateral dichotomy differs from the mammalian pallium, where dorsal and lateral sensory pathways both target a morphologically uniform structure, the neocortex.

The sauropsid LCA may have resembled the modern tuatara, which, unlike any other studied reptile, has a thin cortex-like structure extending from the dorsal cortex through the DVR (Fig. 3, Sauropsid LCA) (32). The DVR may have originated through a tangential expansion of ventricular zone progenitors analogous to neocortical progenitor expansion in mammals, perhaps accompanied by the origin of a vast, associative nidopallium (Fig. 3, brown arrow). This expansion may have resulted in more neurons and more neural processing power for the laterally targeted sensory channels.

The archosaur LCA

We suggest that the archosaur LCA closely resembled modern alligators by pallial morphology (Fig. 3, Archosaur LCA). The archosaur LCA differs from the sauropsid LCA by possessing many more neurons that pack the DVR from the ventricle to the brain surface, but the spatial relationships of the principal neuronal classes are broadly similar. Evolutionary mechanisms of cell number increase are well understood from comparative studies in mammals (26, 33, 34). Archosaurs might have similarly increased neuronal production by prolonging symmetric stem cell divisions or by further elaborating neuronal stem and progenitor cells into distinct ventricular and subventricular zones (27, 35).

The bird LCA

The transition to the LCA of birds involved another burst of increased neurogenesis (13, 26) and a major developmental transformation: the modification of the laminated dorsal cortex into the nuclear Wulst (Fig. 3, Bird LCA). The dorsal cortex and Wulst contain the same core set of cell types but are organized into very different structures (17, 18). For example, alligator dorsal cortex input cells form a superficial cortical layer, but in birds the input population is sandwiched between IT cells and output cells. The morphogenetic mechanisms generating these divergent architectures are unknown.

The mammal LCA

Efforts to describe neocortex origins have invoked the transformation of a reptile-like pallium into the neocortex (9). However, mammals are not thought to have evolved from animals resembling modern sauropsids (29). The transformations leading to the mammalian neocortex need to be considered in the context of the amniote LCA and not the sauropsid lineage. The pallium in the amniote LCA may not have had either a DVR or a cortex, but would later give rise to both architectures (36).

Cell type distribution across the tangential extent of the neocortex is highly uniform, with major cell classes distributed in thin layers extending through the neocortical plate (Fig. 3, Mammal LCA) (19). In contrast, neuronal cell types in the DVR and dorsal cortex are found in spatially segregated territories (10, 1618). These anatomical differences suggest that the mechanisms of cell type specification may differ between mammals and sauropsids. Sauropsid pallial cell types may be specified by spatial information and arise from fate-restricted ventricular zone territories, as demonstrated for the mesopallium and the nidopallium (17, 25). In mammals, a temporal inside-out progression generates the neocortical excitatory neurons (37). The evolutionary origin of the neocortex likely entailed reorganization of the pallial cell type developmental lineages and was not accomplished through elaboration of the reptile cortex.


Natural selection acts on behavior, the functional output of neural circuitry. In this light, the emerging picture from comparative neuroanatomical studies makes sense: The core circuits of the amniote pallium are conserved, but pallial architectures and the spatiotemporal regulation of cell type specification are not (Fig. 4). These latter levels of biological organization can shift over evolutionary time, so long as sensory information gets to the pallium, gets integrated, and elicits an appropriate output. Some transcription factor molecules, such as the hypothesized input cell determinants, exhibit a conserved correlation with defined neuronal cell types across species. We suggest that these factors are evolutionarily maintained together at the cell type level because they collaborate to control the circuits essential for behavior.

Fig. 4 Pallium evolution resembles an hourglass.

A field of embryonic neuronal progenitors (gray cells, bottom center) can be developmentally patterned such that pallial cell types arise from all parts of the neuroepithelium, as in mammals (bottom left), or from restricted parts of the neuroepithelium as in sauropsids (bottom right). Pallial cell types are specified to form conserved connections (middle of hourglass), possibly through conserved transcriptional regulators (see Fig. 2), but migrate to form clade-specific cortical or nuclear architectures (top).


What is the avian homolog of the six-layered neocortex? Put simply, there isn’t one because the neocortex, as a structure, is an evolutionary novelty of mammals. This statement does not imply any paradox. Every morphological character must have a beginning and lack a homolog in other lineages, because the alternative is that the character has always existed. For this reason, it is essential that we refer to levels of homology and recognize that nonhomologous structures can share homologous features (3, 4). The mammalian neocortex and avian pallium share such features: their cell types and circuitries.

An additional level of homology, that of embryological territories, is often invoked to describe the relationships of amniote pallial structures. One hypothesis states that structures are homologous on the basis of their regional developmental origins in the embryo, and that the sauropsid DVR is homologous with the mammalian claustrum and amygdala rather than with the neocortex (5). Several recent studies have embraced this alternative view of pallial evolution (16, 25, 26). We emphasize that developmental origin is one of many features that can be weighed to infer homology of two similar characters. Homologous characters can be generated by nonhomologous mechanisms and from nonhomologous progenitors (3, 4, 7). Nonetheless, concordance between two or more levels, such as between connections and gene expression, can strengthen a claim of homology.

The idea that pallial cell types are rearranged across amniotes has met with some resistance (38). However, there are examples outside of amniotes in which conserved neuronal cell types were driven to adopt different architectures by behavioral and ecological pressures. Goldfish provide one example. In their hindbrains, gustatory secondary sensory neurons and motor neurons of the oral musculature form a multilayered vagal lobe, rather than separate nuclei as in amniotes (39, 40). This highly derived architecture, which places sensory input and motor output neurons in a columnar registration, likely mediates the specialized ability of these scavengers to sort food from gravel in their mouths (39).

By analogy, clade-specific behavioral adaptations may be responsible for the striking diversification of amniote pallial anatomies. Even among reptiles, extensive pallial diversity is evident. In tuataras, the dorsal cortex transitions into a cortex-like DVR (32). The cerebral cortex in some snakes and lizards is splintered into short, overlapping segments (18). Alligator and turtle cortices possess some degree of cell type layering, but the relative arrangement appears to be reversed (10, 16, 18). Finally, there is the variability across DVR organizations (1, 16, 18). Relative to other brain structures, pallial architecture is unusually plastic during evolution. Reptiles provide a venue to investigate the selective pressures and developmental mechanisms that reorganize pallial cell types and circuitries in evolution.

References and Notes

Acknowledgments: We thank M. Coates for comments. Funding: The Molecular and Cellular Biology Training Program (T32 GM007183) and the Developmental Biology Training Program (HD055164). Competing interests: None declared.

Stay Connected to Science

Navigate This Article