Conversion of object identity to object-general semantic value in the primate temporal cortex

See allHide authors and affiliations

Science  18 Aug 2017:
Vol. 357, Issue 6352, pp. 687-692
DOI: 10.1126/science.aan4800

Faulty remembrance of objects past

The primate brain analyzes visual input along the ventral processing stream to extract the identity of an object. The final stage of this stream, the perirhinal cortex, plays a crucial role in object recognition. Tamura et al. systematically biased the judgments of monkeys in an old-new object recognition task by using either optogenetic or electrical stimulation. The monkeys judged an encountered object as familiar when the stimulation site was in a hotspot where memory neurons were clustered. However, at the hotspot's fringe region, where neurons lost selective responses to the learned objects, electrical microstimulation led the monkeys to mistakenly judge an object as never seen before.

Science, this issue p. 687


At the final stage of the ventral visual stream, perirhinal neurons encode the identity of memorized objects through learning. However, it remains elusive whether and how object percepts alone, or concomitantly a nonphysical attribute of the objects (“learned”), are decoded from perirhinal activities. By combining monkey psychophysics with optogenetic and electrical stimulations, we found a focal spot of memory neurons where both stimulations led monkeys to preferentially judge presented objects as “already seen.” In an adjacent fringe area, where neurons did not exhibit selective responses to the learned objects, electrical stimulation induced the opposite behavioral bias toward “never seen before,” whereas optogenetic stimulation still induced bias toward “already seen.” These results suggest that mnemonic judgment of objects emerges via the decoding of their nonphysical attributes encoded by perirhinal neurons.

We daily encounter hundreds of visual objects and instantaneously evaluate whether they are safe or dangerous, friendly or hostile, delicious or toxic, or never seen before. In primates, visual input is analyzed along the ventral stream into constituent features to extract the visual identity of the object by which the representation in long-term memory is reactivated (15). The perirhinal cortex, the final stage of the stream, plays crucial roles in visual object discrimination (6) as well as in visual memory (4, 7, 8). Perirhinal neurons represent multiple attributes of objects, including complex combinations of visual features, linking of dissimilar objects, and object-reward association (4, 6, 7, 9). Notably, through learning, nearby perirhinal neurons often respond to physically dissimilar memorized objects (7, 10) (Fig. 1A).

Fig. 1 Optogenetic manipulation in macaque monkeys.

(A) Object coding in the ventral stream. In early visual cortices, nearby neurons selectively respond to objects with similar orientation. By contrast, in the perirhinal cortex, nearby neurons selectively respond to physically dissimilar complex objects. D, dorsal; V, ventral; A, anterior; P, posterior. (B) Behavioral task. Monkeys judged whether a cue object was one of the memorized OLD objects or a trial-unique NEW object. In half of the trials, the target perirhinal region was optogenetically stimulated for 600 ms from the cue onset. Inset: Lateral view of monkey brain. (C) Cue objects and various levels of visual NEW/OLD valence (19). (D to F) Neuronal responses to cue objects. (D) Raster plot for different NEW/OLD valence levels. Left, OLD objects; right, NEW objects; gray area, cue period. (E) Peristimulus time histogram of the neuron in (D). Top, responses to the best object; middle, responses to the three best and worst OLD objects and to NEW objects (bin width, 25 ms); bottom, rank-ordered responses to OLD and NEW objects (19). (F) Population responses across valence levels (n = 10 neurons, gray lines) (19). Black lines are averages. Colors correspond to those in (E). (G) Magnetic resonance (MR) image of a brain with the glass-coated optrode penetrated. Cartoon at top shows the optogenetically transduced area (rs, rhinal sulcus; sts, superior temporal sulcus; amts, anterior-medial temporal sulcus; ots, occipital-temporal sulcus). L, lateral; M, medial. In the image, arrowheads denote electrolytic lesions for registration with histological sections (19); the dashed square is the area shown in (I) to (K). (H) In vivo fluorescence measurements along the track in (G). (I) Fluorescent histological section corresponding to (G). (J) Bright-field image of (I). (K) Nissl-stained section adjacent to (I). (L to O) Immunohistochemical staining. Yellow and white arrowheads denote ChR2-EYFP–positive and GAD67-positive neurons, respectively. Scale bars, 5 mm (G), 2 mm [(I) to (K)], 20 μm [(L) to (O)].

By contrast, in early sensory areas such as the primary visual (V1) and middle temporal (MT) areas, principal neurons with selective responses to a similar physical attribute of objects (e.g., orientation or motion direction) are spatially clustered as a functional column (11, 12). Microstimulation of the clustered neurons has revealed that the physical attribute is decoded to drive sensory percepts associated with the attribute (13, 14). The apparent differences in the coding scheme between early visual areas and the perirhinal cortex raise the issue of what attributes the perirhinal neurons convey. Does the perirhinal cortex merely progressively relay physical attributes of object identity, such that only the sensory percept of objects is decoded for behavior? Or are nonphysical attributes that are encoded in perirhinal neurons also decoded under an appropriate behavioral context? Here, we addressed these two alternatives within a visual recognition paradigm, focusing on “learned” (a nonphysical attribute). We predict that if the first alternative is correct, then the experimentally excited cluster of neurons coding physically dissimilar objects would cause a subject to judge a presented object as novel, or the subject would not be able to make a judgment because intermingled physical attributes of memorized objects are activated; if the second alternative is correct, a subject would judge a presented object as seen before. To test these predictions, we optogenetically enhanced the output of perirhinal neurons (1517), uncovering the causal link between the mnemonic coding of perirhinal neurons and the behavioral impact of their activities.

Monkeys were trained to discriminate whether a briefly presented (50 ms) visual cue was one of the previously memorized (OLD) objects or a trial-unique (NEW) object presented about once in a month (i.e., once in every 12,000 trials) (Fig. 1, B and C). To evaluate behavioral changes near the psychophysical threshold for OLD/NEW judgment, we obtained a series of perceived NEW/OLD valence of objects by masking cue objects with different levels of pixel-based random-dot dynamic noises (18, 19) (Fig. 1C). Single-unit recordings (Fig. 1, D to F) revealed that the response amplitudes of perirhinal neurons degraded as the OLD valence levels decreased (P = 2.84 × 10−8, Kruskal-Wallis test; Fig. 1F, top). However, their object selectivity was maintained across different valence levels (P = 0.691, Kruskal-Wallis test; Fig. 1F, middle and bottom), suggesting the recruitment of these neurons for object recognition near the psychophysical threshold.

We expressed Channelrhodopsin-2 (ChR2) in a population of excitatory neurons in the perirhinal cortex under the calcium/calmodulin-dependent protein kinase IIα (CaMKIIα) promoter (2022) by injecting an adeno-associated virus vector (AAV5-CaMKIIα-ChR2-EYFP, where EYFP denotes enhanced yellow fluorescent protein) (fig. S1, A to D). In vivo fiber optic measurement (23) confirmed ChR2 expression around the location of virus injection (Fig. 1, G and H); subsequent histological analyses confirmed that ChR2 was expressed predominantly in excitatory neurons (Fig. 1, I to O, and fig. S1E). Using this in vivo measurement, we confirmed the expression of ChR2 in the respective behaving monkeys (fig. S2). Neurons were effectively excited by repetitive short-pulse optogenetic stimulation with 473-nm light optimal for ChR2 (24), with more than 50% efficacy of firing at 200-Hz stimulation (fig. S3).

Optogenetic stimulation during the task, relative to sham illumination, biased monkeys’ choices toward OLD objects across different NEW/OLD valence levels (Fig. 2, A to E), as indicated by significant leftward shifts of psychometric curves (13, 19). In 36 of 73 experimental sessions, optogenetic stimulation induced significant bias toward OLD choices (positive equi-NEW/OLD valence; P = 1.33 × 10−67 in χ2 test against false positive rate of 5%), and there were no cases of significant bias toward NEW choices. As a result, the distribution of equi-NEW/OLD valence was significantly biased toward the positive direction (P = 2.97 × 10−12, Wilcoxon signed-rank test; Fig. 2E). This behavioral bias toward OLD choices was observed in the respective monkeys (fig. S4, A and B) (19). No behavioral changes were induced by stimulation with 594-nm light that was ineffective for ChR2 excitation (24) (P = 0.129, Wilcoxon signed-rank test; Fig. 2, F to H), nor by 473-nm light stimulation to neurons expressing only EYFP but not ChR2 (P = 0.938, Wilcoxon signed-rank test; fig. S5).

Fig. 2 Optogenetic stimulation of perirhinal neurons biases monkeys’ recognition judgment.

(A and B) Schematic of optogenetic stimulation experiments. (A) Two optical channels were used for optogenetic stimulation as well as sham illumination to prevent the monkeys from using external cues to discriminate between stimulated and nonstimulated trials. (B) The order of stimulated and nonstimulated (sham-illuminated) trials was randomized. (C and D) Representative psychometric curves from two monkeys stimulated with 473-nm light optimal for exciting ChR2. P values show statistical significance of the horizontal shift (equi-NEW/OLD valence) and slope change in each experimental session. (E) Distribution of horizontal shifts induced by 473-nm light stimulation (n = 73 experimental sessions). Positive values show that the optogenetic stimulation biases monkeys’ judgments toward the OLD choices. Experimental sessions showing a significant horizontal shift (P < 0.05) are indicated by colored bars (19); the P value is the statistical significance of distribution shift from zero, evaluated by Wilcoxon signed-rank test. (F and G) Representative psychometric curves from two monkeys stimulated with 594-nm light. (H) Distribution of horizontal shifts induced by 594-nm light stimulation (n = 12 experimental sessions). Stimulation with 594-nm light was less effective than stimulation with 473-nm light (P = 6.05 × 10−6, Wilcoxon rank sum test).

We further quantified the stimulation-induced bias toward OLD choices. The slope of psychometric curves that reflected the performance of monkeys’ discrimination was slightly flattened by stimulation (P = 0.00136, Wilcoxon signed-rank test; significant slope change in 9 of 73 experimental sessions, P = 0.00407 in χ2 test against false positive rate of 5%; fig. S4, A and B). However, the effect size of slope flattening was significantly smaller than that of leftward shifts (P = 2.60 × 10−10, χ2 test for numbers of experimental sessions with significant horizontal shift and slope change; fig. S4, A and B) (19). The optogenetic stimulation indeed induced behavioral bias toward OLD choices for judgments of both NEW and OLD objects (NEW, P = 8.59 × 10−11; OLD, P = 3.37 × 10−9; Wilcoxon signed-rank test with Bonferroni method; fig. S4C). Moreover, the stimulation-induced bias toward OLD choice did not accompany the increase in the total rewarded rate (fig. S4D), indicating that the behavioral change toward OLD choice occurred without any incentive for rewards (see fig. S4, E to H, for the time-series analysis of stimulation effects) (19).

We next examined whether and how these stimulation-induced behavioral changes were related to neuronal responsiveness to learned objects at the stimulation sites. First, we confirmed that the nearby perirhinal neurons with high stimulus selectivity responded to learned objects distinct from each other (fig. S6). Then, we evaluated the neuronal representation at each stimulation site by calculating the density of OLD object–responsive neurons (DOLD) and NEW object–responsive neurons (DNEW) for each site. The optogenetic stimulation–induced bias did not depend on the relative dominance of DOLD and DNEW at the stimulation site (regression, Y = –1.62X + 2.88; Fig. 3A) (19).

Fig. 3 Relationship between neuronal activity and stimulation-induced behavioral bias.

(A) Main panel: Scatterplot of the behavioral effects of optogenetic and electrical stimulations as a function of normalized difference in the density of OLD object–responsive neurons (DOLD) to that of NEW object–responsive neurons (DNEW) around the stimulation site (19). Blue circles, optogenetic stimulation (n = 73 experimental sessions; see also Fig. 2E); gray circles, electrical stimulation (n = 39 experimental sessions; see also fig. S7); blue and black lines, regression lines for optogenetic and electrical stimulations, respectively. The slope difference between the two regression lines was statistically significant (P = 6.68 × 10−8, interaction of analysis of covariance), indicating dissociable behavioral effects between optogenetic and electrical stimulations. Inset of the scatterplot: Mean equi-NEW/OLD valence as a function of normalized difference between DOLD and DNEW (bin width, 0.05) in optogenetic and electrical stimulations. Panels at left and right: Pairs of psychometric curves of optogenetic or electrical stimulations at the sites showing similar object-coding properties (squares and triangles). (B) Comparison of regression parameters. *P < 0.001 (difference from zero, corrected for multiple comparisons); †P < 0.05 (corrected for multiple comparisons). (C) Stimulation effects categorized according to the relative dominance of DOLD (electrical, n = 21; optogenetic, n = 47) and DNEW (electrical, n = 12; optogenetic, n = 21) (P = 1.89 × 10−6, F = 25.7, interaction of two-way ANOVA). *P < 0.05 (difference from zero, t test with Bonferroni method); †P < 0.05 (t test with Bonferroni method).

We then tested the effect of conventional electrical stimulation. In contrast to the optogenetic stimulation, the electrical stimulation induced biases toward NEW choices as well as OLD choices (fig. S7). The induced choice bias was proportional to DOLD at the stimulation site (regression, Y = 9.84X – 1.54; Fig. 3A). Accordingly, optogenetic and electrical stimulations induced doubly dissociable behavioral impacts on object recognition (P = 6.68 × 10−8, interaction of analysis of covariance); electrical stimulation had a significantly positive slope (P = 8.2 × 10−4, t test with Bonferroni method) but optogenetic stimulation did not (P = 0.160). Optogenetic stimulation had a significantly positive intercept (P = 1.62 × 10−14) but electrical stimulation did not (P = 0.127) (Fig. 3B). We confirmed that this double dissociation was observed in the respective monkeys (fig. S8). Note that similar results were obtained when the induced choice biases were categorized into two groups according to the relative dominance of DOLD and DNEW [Fig. 3C; P = 1.89 × 10−6, interaction of two-way analysis of variance (ANOVA)]. We also confirmed similar behavioral bias at a lower frequency of electrical stimulation (fig. S9).

In the object-object association memory task, neurons that code linking of objects are not uniformly distributed in the perirhinal cortex but are clustered at the anterior region of the longitudinal stripe of the perirhinal cortex (25). Thus, we examined (i) whether neurons that memorize OLD objects in this recognition task were also clustered in a restricted region along the longitudinal axis of the perirhinal cortex; and, if so, then (ii) whether the stimulation-induced biases in recognition differed along the longitudinal axis (Fig. 4A). The amplitudes of neuronal responses to OLD objects (Max response) and DOLD were higher in the anterior region (Fig. 4B and fig. S10B; see fig. S10A for spatial distribution of behavioral effects). An unsupervised cluster analysis with four parameters (DOLD, DNEW, Max response, and the longitudinal coordinate) revealed two clusters that were mainly segregated along the longitudinal axis (fig. S11, A and B) (19). Electrical stimulation exerted the opposite behavioral effects along the axis: Behavioral biases toward OLD and NEW choices were induced by stimulation of the anterior and posterior subregion, respectively (Fig. 4C, lower panels, P = 0.0453 for anterior and P = 2.79 × 10−4 for posterior subregion, Wilcoxon signed-rank test with Bonferroni method; P = 9.64 × 10−6 for difference in behavioral effects between anterior and posterior subregions, Wilcoxon rank sum test with Bonferroni method) (see fig. S11, C and D, for the respective monkeys; see fig. S12, C to F, for the time-series analysis of stimulation effects) (19). In sharp contrast, optogenetic stimulation showed the behavioral bias toward OLD choices consistently in both the anterior and posterior subregions (Fig. 4C, upper panels, P = 1.51 × 10−7 for anterior and P = 8.46 × 10−5 for posterior subregions, Wilcoxon signed-rank test with Bonferroni method); the difference between the anterior and posterior subregions was not statistically significant (P = 1.00, Wilcoxon rank sum test with Bonferroni method; see also fig. S11, E to J). Accordingly, the behavioral effects of optogenetic and electrical stimulations were significantly different in the posterior subregion but not in the anterior subregion (Fig. 4C, P = 3.39 × 10−8 for posterior, P = 1.00 for anterior, Wilcoxon rank sum test with Bonferroni method) (see also figs. S13 and S14). Neither optogenetic nor electrical stimulations showed significant behavioral effects when applied at the anterior and posterior peripheral ends (Fig. 4D). Notably, neuronal responses to OLD objects were higher in the anterior focal spot, and stimulus selectivity to OLD objects abruptly decreased at the adjacent fringe region.

Fig. 4 Behavioral impacts of neuronal output from subregions of the perirhinal cortex.

(A) Schematic for examination of behavioral impact of optogenetic or electrical stimulation on object recognition along the longitudinal axis of rostral area 36 (36r). A red arrow denotes the longitudinal axis of 36r (19). (B) Spatial distribution of the response amplitude (Max response) of recorded neurons. AP, anterior-posterior coordinate from interaural line; LM, lateral-medial coordinate from midline; gray lines, location of lip of rhinal sulcus in each hemisphere; circles and squares, location of stimulation sites in clusters 1 and 2, respectively (see fig. S11 for determination of clusters). (C) Behavioral impacts of optogenetic or electrical stimulation of each subregion within 36r (optogenetic, n = 46 and 27 experimental sessions for clusters 1 and 2; electrical, n = 17 and 22 experimental sessions for clusters 1 and 2). P values denote statistical significance evaluated by Wilcoxon signed-rank test with Bonferroni method. (D) Distribution of stimulation-induced choice bias along the longitudinal axis. The longitudinal coordinate of 0 mm was set at the location of the anterior commissure on MR images. Neuronal indices of Max response (green) and Selectivity (purple) permit comparison between changes in behavioral effects and changes in neuronal responses. *P < 0.05 (difference from zero, t test with Bonferroni method); †P < 0.05, uncorrected. Inset: Transition of relationship between behavioral impacts in electrical and optogenetic stimulation. Shown are means ± SEM of equi-NEW/OLD valence in stimulation sites, grouped along the axis. Asterisks denote significant deviation from the origin of both coordinates (95% confidence ellipse with Bonferroni method).

Our results show that optogenetic activation of projection neurons in the perirhinal cortex induced a systematic increase in monkeys’ OLD choices, irrespective of the local object-feature coding of neurons at the stimulation site. These results suggest that activation of perirhinal projection neurons produces an object-general “learned” signal. It is unlikely that nonphysiological effects of the optogenetic stimulation could explain the observed effects. First, we used optogenetic stimulation parameters comparable to those used in previous studies (19, 26, 27). Second, our findings did not result from nonselective energy transfer or heating (Fig. 2, F to H, and fig. S5). Third, using these parameters, we observed a long-tailed distribution of behavioral effects on one side with the peak at near-zero (Fig. 2E); this skewed distribution has frequently been observed in previous electrical stimulation studies that examined behavioral effects near the threshold for effective versus ineffective stimulation (13). Our electrical stimulation parameters were comparable to those used in previous studies in the inferior temporal cortex (18, 28) but stronger than those used in early visual areas (13, 14), presumably reflecting an increase in the effective current amplitude for inducing behavioral effects along the ventral visual stream (29) and/or inhibitory-dominant circuits of the perirhinal cortex (30, 31).

Our findings also shed light on a hypothesis for the mechanism underlying object recognition in the perirhinal cortex: When encountering a “learned” object, a group of perirhinal neurons are activated and produce an object-general “learned” signal by referring to the specific memory of that object. When the total output of perirhinal neurons is enhanced to reach a given threshold by this reference, or by optogenetic or electrical stimulation, monkeys judge the object as OLD. On the other hand, when the total output does not reach the threshold, monkeys judge the object as NEW (19). These observations suggest an architectonic principle of the brain by which the nonphysical attributes of object information are processed for semantic reasoning.

Supplementary Materials

Materials and Methods

Supplementary Text

Figs. S1 to S14

References (3298)

References and Notes

  1. See supplementary materials.
Acknowledgments: We thank K. Deisseroth for optogenetic constructs; R. J. Samulski and the University of North Carolina Vector Core for virus vectors; K. Jimura and T. Matsui for helpful discussions; Y. Ohashi for management of the virus experiment facility; K. W. Koyano for the MR imaging protocol; and A. Fukuda, T. Watanabe, K. Kakizawa, and K. Date for experimental assistance. Macaques used in this research were provided by NBRP “Japanese Monkeys” through the National BioResource Project of the Ministry of Education, Culture, Sports, Science and Technology (MEXT). Supported by MEXT and Japan Society for the Promotion of Science (JSPS) KAKENHI grants 17H06161 and 24220008 (Y.M.), 16H01281 and 17K07062 (M.T.), 26890007 (R.S.), and 15H01419 and 17H02219 (T.H.); CREST, Japan Science and Technology Agency (Y.M.); AMED-CREST, Japan Agency for Medical Research and Development (Y.M.); Brain Sciences Project of the Center for Novel Science Initiative, National Institutes of Natural Sciences, grant BS271006 (M.T.); Brain/MINDS from AMED (T.H.); the Strategic Research Program for Brain Sciences from AMED (T.H.); and JSPS Research Fellowships for Young Scientists grant 265926 (K.M.). All data necessary to support this paper’s conclusions are available in the supplementary materials. Author contributions: K.T. contributed study design, data acquisition (electrophysiology, optogenetic/electrical behavioral intervention, histology), and data analysis; M.T. contributed data analysis; R.S. contributed data acquisition (histology) and data analysis; T.T. contributed data acquisition (electrophysiology and histology); T.H. contributed study design and data analysis; K.M. contributed data analysis; Y.M. contributed study design, data analysis, and overall supervision; and all authors contributed to writing the manuscript.
View Abstract

Stay Connected to Science

Navigate This Article