Research Article

Flexible recruitment of memory-based choice representations by the human medial frontal cortex

See allHide authors and affiliations

Science  26 Jun 2020:
Vol. 368, Issue 6498, eaba3313
DOI: 10.1126/science.aba3313

The adaptive human frontal cortex

Flexibly switching between different tasks is a fundamental human cognitive ability that allows us to make selective use of only the information needed for a given decision. Minxha et al. used single-neuron recordings from patients to understand how the human brain retrieves memories on demand when needed for making a decision and how retrieved memories are dynamically routed in the brain from the temporal to the frontal lobe. When memory was not needed, only medial frontal cortex neural activity was correlated with the task. However, when outcome choices required memory retrieval, frontal cortex neurons were phase-locked to field potentials recorded in the medial temporal lobe. Therefore, depending on demands of the task, neurons in different regions can flexibly engage and disengage their activity patterns.

Science, this issue p. eaba3313

Structured Abstract

INTRODUCTION

Decision-making in complex environments relies on flexibly combining stimulus representations with context, goals, and memories. A central component of cognitive flexibility is to selectively retrieve information from memory and utilize the retrieved information to make decisions. The medial frontal cortex (MFC) plays a critical role in this process by representing task sets, context, and outcomes. During decision-making, the MFC is thought to selectively engage memory retrieval by representing memory-based choices and mediating interactions between the frontal lobes and the hippocampus and amygdala (HA) through phase-locking of MFC activity to oscillations in the HA. It remains unknown what features of decisions and context are represented in the human MFC and what functional interactions between the MFC and HA mediate dynamic memory retrieval during a task.

RATIONALE

We recorded single neurons and local field potentials in the human MFC and HA in patients implanted with depth electrodes. Subjects switched between two tasks: recognition memory and categorization. To identify signatures of task demands, we compared the strength of encoding of stimulus familiarity, category, and choices between tasks and tested whether decoders trained in one task generalized to the other task. Such cross-task generalizability would indicate abstract representations of the underlying variables. We hypothesized that this approach would reveal neural signatures of the representations and functional interactions that permit memory-based decisions.

RESULTS

We recorded from 1430 single neurons in the HA and MFC [dorsal anterior cingulate cortex (dACC) and the pre-supplementary motor area (pre-SMA)] across 13 patients. Subjects made “yes” or “no” decisions using button presses or saccades (eye movements) to indicate whether an image was novel or familiar, or whether an image belonged to a given visual category. Instructions were given before each block of trials, explaining the task and response modality to use (i.e., task set). Examining the underlying neural representations at the single-neuron and population levels revealed the following: (i) Cells in the MFC represented task set during baseline periods. These contextual signals emerged rapidly after a task switch and generalized across all response and task-type combinations in the MFC but not the HA. (ii) The strength and geometry of representations of familiarity were task-insensitive in the HA but not in the MFC. The responses of these memory-selective cells were a reflection of memory strength rather than decisions about the memory. (iii) The visual category of stimuli was represented more strongly during the memory task in both the MFC and HA. This encoding of category generalized across tasks fully in the HA but not the MFC. (iv) Choices in both tasks were most strongly represented by cells in the MFC. This choice representation differed in its population-level geometry between the two tasks but was insensitive to response modality (button press or saccade). One subset of MFC cells signaled only memory-based choices, and these cells signaled decisions about the memory. (v) MFC cells phase-locked their activity to theta-frequency band oscillation in the HA preferentially in the memory task, with memory-choice cells also phase-locking in the gamma-frequency band. The strength of this interareal phase-locking in both frequency bands of the MFC cells that signaled memory-based choices was predictive of behavior.

CONCLUSION

We leveraged the opportunity to record from single neurons in humans to identify representations of choices, task sets, stimulus category, and familiarity in the human MFC and HA. We found that neuronal populations within the MFC formed two separate decision axes: one for memory-based decisions and another for categorization-based decisions. MFC-HA theta-frequency functional connectivity was selectively enhanced during memory retrieval. This work reveals a neuronal mechanism in the human brain whereby oscillation-mediated coordination of activity between distant brain regions and accompanying changes in strength of representation and/or geometry implements task-dependent retrieval of memory.

Flexible representations of choices in the human frontal lobe.

(A) Recording locations. LFP, local field potential. (B) Population response of all recorded neurons (left) and example of a cell signaling memory-based choices (right). (C and D) Representational geometry analysis reveals that different subspaces are used by the two tasks, establishing a memory-specific decision axis. (E) Theta- and gamma-band coherence of MFC choice cells with HA LFPs increased during the memory task.

Abstract

Decision-making in complex environments relies on flexibly using prior experience. This process depends on the medial frontal cortex (MFC) and the medial temporal lobe, but it remains unknown how these structures implement selective memory retrieval. We recorded single neurons in the MFC, amygdala, and hippocampus while human subjects switched between making recognition memory–based and categorization-based decisions. The MFC rapidly implemented changing task demands by using different subspaces of neural activity and by representing the currently relevant task goal. Choices requiring memory retrieval selectively engaged phase-locking of MFC neurons to amygdala and hippocampus field potentials, thereby enabling the routing of memories. These findings reveal a mechanism for flexibly and selectively engaging memory retrieval and show that memory-based choices are preferentially represented in the frontal cortex when required.

Behavior in complex environments requires decisions that flexibly combine stimulus representations with context, goals, and memory. Two key aspects of cognitive flexibility are the selective utilization of relevant information depending on task demands and the retrieval of information from memory, when needed (1). We are beginning to understand the neural mechanisms that underlie flexible decisions in the case of perceptual decision-making (24), with evidence for both early gating, mediated by top-down attention (5), and late selection of relevant features in the prefrontal cortex (3). In contrast, little is known about the decision mechanisms that also depend on associated category knowledge and memory. In particular, it is not clear how memory retrieval is selectively engaged when decision-relevant information needs to be actively searched for in memory (68).

The medial frontal cortex (MFC) is critical for complex behavior and registers cognitive conflict, errors, and choice outcomes (911). It supports flexible decision-making in two ways: (i) by representing task sets (1214) and context (15), and (ii) by selectively engaging memory retrieval through functional interactions with other brain areas (1618), specifically the hippocampus (1921) and amygdala (22, 23). A mechanism that facilitates such interareal interactions is phase-locking of MFC activity to osciollations in the hippocampus or amygdala. This mechanism has been extensively investigated in rodents during spatial behavior (2426) and fear conditioning (27, 28), but its broader function remains poorly understood (29), particularly in humans. Similarly, human neuroimaging studies indicate that the MFC is involved in memory search (8, 18, 3034) and that patterns and level of activity and connectivity assessed by functional magnetic resonance imaging (fMRI) vary as a function of retrieval intentionality (3538). It is not yet known what features of decisions and context are represented in the human MFC, whether memory retrieval selectively engages synchrony between the MFC and the hippocampus and/or amygdala, and whether synchrony can be engaged dynamically when required. This lack of knowledge stands in stark contrast to the patent behavioral ability of humans to flexibly recruit memory processes in everyday life (39, 40) and to our detailed knowledge of memory representations in the human hippocampus and amygdala, where cells represent aspects of declarative memories, such as the familiarity and the identity of a stimulus (4143). To address these open questions, we used simultaneous recordings of single neurons and local field potentials (LFPs) in the human MFC, hippocampus, and amygdala.

Task and behavior

We recorded from 1430 single neurons across four brain areas (Fig. 1, C and D; see table S1; 33 sessions in 13 subjects): n = 203, 460, 329, and 438 neurons from anterior hippocampus (HF), amygdala (AMY), dorsal anterior cingulate cortex (dACC), and pre-supplementary motor area (pre-SMA), respectively. For brevity, we refer to HF and AMY together as HA (n = 663 neurons) and to dACC and pre-SMA together as MFC (n = 767 neurons).

Fig. 1 Task, electrode locations, and behavior.

(A) Task structure. A session consisted of eight blocks of 40 trials. The task switched with each block (blue = categorization, red = memory), and the response modality switched halfway through each block (saccade or button press, randomly assigned at the beginning of the block). The subject was instructed about the task at the beginning of each block (purple arrows) and how to respond at the beginning and halfway points of each block (green arrows). (B) Example of screens shown to subjects for two example trials. (C and D) Electrode locations. Each dot is the location of a microwire bundle in one subject. Coordinates are in Montreal Neurological Institute (MNI) 152 space. (E and F) Eye tracking data from one session from the button press (E) and eye movement (F) trials. (G) Reaction times as a function of task across all sessions (memory, μ ± SEM, 1.27 ± 0.02 s; categorization, 0.90 ± 0.02 s; P = 7.6 × 10−228, two-sample KS test). (H) Memory performance improves over the course of the experiment (β = 0.56, P = 8.42 × 10−130, logistic mixed effects model). See fig. S1 for an extended summary of the behavior.

Human subjects viewed a sequence of 320 images, grouped into eight blocks of 40 images each, in each session (Fig. 1, A and B). At the beginning of each block, subjects were instructed which decision to make and which response modality to use to communicate their decision. Subjects made a “yes” or “no” decision for each trial to indicate whether an image belonged to a given visual category (“categorization task”) or whether an image had been seen before in the task or not (“memory task”). No feedback was provided (see Materials and methods section for details on the task). Each image shown belonged to one of four visual categories: human faces, monkey faces, fruits, or cars. In each block, half of the images shown were repeated and half were novel (except in the first block, in which all images were novel).

Subjects indicated choices using either saccades (leftward or rightward eye movement) or button press while maintaining fixation at the center of the screen (Fig. 1, E and F; mean ± SD, 94 ± 15% of all gaze positions fell within the image shown). Reaction times (RTs) were significantly longer in the memory task than in the categorization task [Fig. 1G, mean RT of 1.48 ± 1.1 s versus 1.19 ± 1.2 s, respectively, P < 1 × 10−20, two-sample Kolmogorov-Smirnov (KS) test, mean ± SD across all trials in a given task]. Subjects performed with an average accuracy of 97 ± 6% versus 71 ± 6% in the categorization and memory tasks, respectively (mean ± SD across n = 33 sessions). This difference in accuracy remained after we matched for RT between the two tasks (96 ± 6% versus 72 ± 8% with matched RTs of 1.23 ± 0.60 s versus 1.24 ± 0.60 s for the categorization and memory task, respectively). Even without RT matching, the initial response in terms of arousal was not different between tasks, as assessed by pupillometry (fig. S1, J to L). In the memory task, accuracy increased as a function of how many times an image had been shown (Fig. 1H, βappearances = 0.56, P < 1 × 10−20, mixed effects logistic regression; also see fig. S1, C and D, for effect of target versus nontarget on memory performance). Subjects had shorter RTs on “yes” (seen before) decisions than on “no” (novel stimulus) decisions in the memory task (fig. S1A, see legend for statistics), as expected from a medial temporal lobe (MTL)–dependent recognition memory task (41). In the categorization task, RT was not significantly different between the two responses (fig. S1A), showing the absence of oddball effects.

Effects of task type and response modality in the MFC

Instructions about the task type and response modality were shown at the beginning of each block (Fig. 1, A and B). Cells showed significant modulation of their firing rate during the baseline period as a function of task type (Fig. 2, A and B, shows an example in pre-SMA). At the single-neuron level, significantly more cells were modulated by task type in the MFC than in the HA: 25% of MFC cells (165/767, 82 in dACC, 83 in pre-SMA; see fig. S2B) versus 12% of HA cells (79/663, 21 in HF, 58 in AMY), χ2 test of proportions, P < 1.5 × 10−6. Similarly, at the population level, population decoding accuracy was significantly higher in the MFC than in the HA [Fig. 2C; 90% versus 70%, respectively; P < 1 × 10−3; true difference (Δtrue) = 20% versus empirical null distribution; see Materials and methods], a conclusion that held regardless of the number of neurons used (fig. S2H). Cells also modulated their activity as a function of response modality during the baseline period (fig. S2E shows an example). As with task-type encoding, significantly more cells encoded response modality in the MFC, and this signal could be decoded with higher accuracy in the MFC than in the HA (14% versus 10% of cells; 84/593 versus 59/586 in the MFC and HA, respectively; 33 in ACC, 51 in pre-SMA, 27 in HF, 32 in AMY; χ2 test of proportions, P = 0.03; population decoding performance 72% versus 64%, Fig. 2D; P < 0.05, Δtrue = 8% versus empirical null distribution); this conclusion held regardless of number of neurons used (fig. S2I).

Fig. 2 Representations of task type and response modality.

(A and B) Example pre-SMA neuron. (B) Average firing rate during the baseline period (−1 to 0 s relative to stimulus onset) for each block for the cell shown in (A). The average baseline firing rate across all blocks of the same type is shown. (C and D) Population decoding of task type (C) and response modality (D). (E) Cross-condition decoding approach. The background color denotes the type of trials that were used to train a given decoder. (F) Cross-response modality decoding of task type from the baseline firing rate of all recorded cells. (G) Cross-task decoding of response modality. (H) Decoding performance as a function of trial number relative to a task type switch (green arrows in Fig. 1A; transitions from categorization to memory and vice versa were pooled). Error bars indicate SD in all panels, with the exception of (B), where they indicate SEM. (I) Baseline decoding of task type for subsequent trials with short reaction times was more accurate than decoding on long reaction time trials. Performance is shown separately for categorization (Cat) and memory (Mem) trials (P = 2 × 10−11 and 7 × 10−13, respectively, Wilcoxon rank sum test). Error bars denote standard error in decoding accuracy across trials (80 trials in each of the four groups). See fig. S2 for additional analyses that break down context effects by specific anatomical regions.

After a task switch, contextual signals emerged rapidly within one to two trials in the new context in the MFC (Fig. 2H). This was not a result of ongoing poststimulus processing, because the task could still be decoded even if only considering the subset of task cells in the MFC, which did not differentiate between the tasks around response time (see fig. S2G). Task-switching costs were also reflected in the subjects’ longer reaction times shortly after a change in task or effector type (fig. S2A). Task-type representations during the baseline period were stronger on trials where the subject subsequently produced a fast response than on those where the response time was slow (Fig. 2I), indicating behavioral relevance. We also tested whether the two types of contextual signals were sufficiently robust to avoid interference with one another, using a cross-condition generalization decoding analysis (44) (Fig. 2E). We first trained a decoder to discriminate task type on trials where the subject was instructed to reply with a button press, and then we tested the performance of this decoder on trials where the subject was instructed to use saccades (and vice versa). The two decoders generalized in the MFC but not in the HA (Fig. 2, F and G). For this reason, we focused on the MFC when conducting the analysis above.

Cross-condition generalization of familiarity and image category

Next, we examined whether the neural representations of image category and familiarity are sensitive to task demands. We assessed two consequences of task demands: generalization across tasks and strength of representations within each task. At the single-unit level, we examined visually selective (VS) cells (42), whose responses are thought to reflect input from high-level visual cortex, and memory-selective (MS) cells (41), whose response signals stimulus familiarity (Fig. 3, A and B, shows examples). Of the HA cells, 40% were visually selective (264/663, 35 in HF and 229 in AMY) and 11% were memory-selective (73/663, 10 in HF, 63 in AMY; 24/73 were old>new selective and 49/73 were new>old selective; see Materials and methods for selection model). In the MFC, 13% (103/767) of the cells were visually selective and 11% (84/767) were memory-selective. First, we performed single-neuron analysis of the selected HA cells. Visual and memory selectivity were present in both the memory and categorization blocks (figs. S3, D, E, H, and I). MS cell responses reflected a memory process: they strengthened over blocks as memories became stronger (fig. S3G), and they differed between forgotten old (false negative, FN) and correctly identified new (true negative, TN) stimuli for both new>old (n = 49) and old>new (n = 24) preferring MS cells (fig. S10F). Furthermore, these memory signals were behaviorally relevant: new/old decoding was significantly weaker in incorrect trials than in correct trials (fig. S10G).

Fig. 3 Representations of image category and familiarity (new versus old).

(A and B) Example cells that (A) represent image category and (B) differentiate between new and old stimuli. (C) Decoding accuracy of image category from all recorded cells was significantly higher in the HA relative to the MFC (Δtrue = 49%, P < 0.001). (D) Decoding of new versus old (ground truth) was similarly accurate in the HA and MFC (Δtrue = 7%, P = 0.13). For new versus old decoding, trials with images of monkeys were excluded, because the recognition performance for these images was at chance (fig. S1B). (E) Population activity of all recorded HA (left) and MFC (right) cells, plotted in 3D using MDS. Individual points show the mean activity of the population for that specific condition. The highlighted plane contains all locations of state space occupied by a given task for the case of fruits versus faces as the binary category distinction (for illustration only; all analysis uses all categories). The geometry of the representation allows for a decoder that is trained on one task to generalize to the other task (see fig. S4C for example decoder hyperplanes). (F) Approach used for the cross-condition generalization analysis. Color indicates task (blue = categorization, red = memory). (Top) We trained a decoder to discriminate between new and old trials on categorization trials and then tested its performance on new and old stimuli encountered during the memory condition (and vice versa). (Bottom) Similarly, a decoder that is trained to discriminate between image categories (in this example, faces versus fruits; all results include all six possible pairs) on categorization trials was tested on memory trials. (G) Cross-condition generalization performance for image category. (H) Cross-condition generalization performance for new versus old. (I) Difference in cross-task generalization decoding accuracy for familiarity and image category between HA and MFC. Difference is computed between the average cross-task performances in each area (i.e., average of memory→categorization and categorization→memory). The null distribution for the average was estimated from the empirical null estimated by shuffling the labels used to train the decoders. For both variables, decoding from HA had significantly greater cross-task generalization performance than decoding from MFC (the difference in both cases is positive and outside of the 95th percentile of the null distribution). (J) Generalization index (see Materials and methods) for memory (two data points on the left) and image category (two data points on the right). For both image category and familiarity, generalization across task was higher in the HA population than in the MFC population (see figure for statistics; Δ, difference).

To answer the question of whether coding for visual and memory information generalizes across tasks, we took a population-level approach (over all single units, without selection). We used decoding performance on individual trials and single-neuron analysis (fig. S4, D and E) to assess whether the neural encoding of visual category and/or familiarity of a stimulus depended on task demands. In both the HA and MFC, image category could be decoded (Fig. 3C, 98 and 49% in the HA and MFC, respectively; chance level = 25%). Category decoding performance was significantly higher in the HA than the MFC (Δtrue = 49% versus empirical null distribution, P < 1 × 10−3). In the HA, the ability to decode category was not significantly different between the two tasks (Fig. 3C, 96% versus 99% in categorization and memory, respectively; Δtrue = 3%, P = 0.25) and could be decoded above chance in both the HF and AMY (fig. S4, A and B). At the single-neuron level, HA neurons encoded significantly more information about category in the memory task (fig. S4D), an effect that decoders were not sensitive to because of saturation. In the MFC, decoding accuracy for image category was significantly higher in the memory task (Fig. 3C, 60% versus 36%, Δtrue = 24% versus empirical null distribution, P < 0.001). Memory was decodable in both the HA and MFC (Fig. 3D, 69% versus 76%, respectively), with no significant difference in decoding accuracy between the two tasks in the HA (Fig. 3D, 67% versus 71% in categorization and memory trials, respectively; Δtrue = 4%, P = 0.3) and significantly better decoding ability in the MFC in the memory task (Fig. 3D, 86% versus 61%; Δtrue = 25% versus empirical null, P = 0.001). Single-neuron analysis confirmed the impression from decoding that the strength of memory signals in the HA was not modulated by task demands, whereas in the MFC it was (fig. S4E). In either task, at the population level, memory decoding in the HA was only possible in the amygdala (fig. S4B). The population-level decoding of memory in the HA was principally a reflection of the signal carried by the MS cells (fig. S10E) and was not due to repetition suppression of VS cells (fig. S10, B and C), because it was not possible to decode familiarity from VS cells alone (fig. S10D).

To gain insight into the geometry of the population-level representations, we assessed whether the decoders trained to report familiarity and the category of the stimuli in one task would generalize to the other task (Fig. 3F schematizes our approach). Cross-task generalizability would indicate that familiarity and visual category are represented in an abstract format (44). First, cross-task generalization performance was greater in the HA than MFC for both image category (Fig. 3, G and I; 98% versus 41%, averaged across the two cross-condition decoding performances; Δtrue = 57% versus empirical null, P < 1 × 10−3) and familiarity (Fig. 3, H and I; 67% versus 55%, Δtrue = 12% versus empirical null, P < 0.05). Second, to help understand the geometry of these neural representations, we projected the average HA and MFC population activity for all possible pairings of familiarity, image category, and task (eight different conditions) into a three dimensional (3D) state space using multidimensional scaling. For illustration purposes, we show this 3D state space for the two image categories (fruits and faces) for which memory performance was the best. In the HA (Fig. 3E, left), the relative positions of a “new face” with respect to an “old face” were preserved across tasks (shown as differently colored planes). The relatively parallel location of the subspace of neural activity occupied by the two tasks permits cross-task generalization for both image category and familiarity. In contrast, in the MFC (Fig. 3E, right), the relative positions of the four conditions were not preserved. This is consistent with the weaker cross-task generalization performance observed in the MFC relative to the HA (Fig. 3, G and H), resulting in reduced generalization indices in the MFC compared with the HA (Fig. 3J; this metric takes into account different levels of within-task decoding accuracy, which is an upper bound for the achievable generalization performance; see the supplementary text section of the supplementary materials for details).

Representation of choice

We next investigated how the subject’s choice (yes or no) is represented by single neurons (Fig. 4A shows examples) and the population. Decoding accuracy for choices was highest in the MFC, with an average population decoding performance of 89% compared with 68% in the HA (Fig. 4B; Δtrue = 19% versus empirical null, P < 1 × 10−3; 61% in AMY and 57% in HF when trained separately; fig. S7G shows this result as a function of number of neurons used). Further single-neuron (fig. S11, A and B) and population (Fig. 4E and fig. S11C) analysis confirmed that the choice signal was significantly stronger in the MFC regardless of selection threshold and quantification method. We therefore first analyzed choice information in the MFC (see below for results in the HA). Choice decoding in the MFC was strongest shortly after stimulus onset, well before the response was made (fig. S7A). To disassociate representation of choice (yes or no) from the representation of ground truth (old or new) during the memory recognition task, we fit a choice decoder to a subset of trials, half of which were correct and half of which were incorrect. The activity of MFC cells predicted choice but not the ground truth at levels significantly different from chance (Fig. 4C; choice decoding is above the 95th percentile of the null distribution, whereas new/old decoding is not; see fig. S7D for this analysis shown over time). Choice could be decoded from the MFC separately for both the correct and the incorrect trials (fig. S5I). As a control for potential confounds due to RT differences between tasks (see fig. S1A), we acquired data from a separate control task in which we eliminated RT differences behaviorally by adding a waiting period (six sessions in five subjects; n = 180 and 162 neurons in the HA and MFC, respectively; see Materials and methods and fig. S6). As in the original task, MFC cells represented the subject’s choice (fig. S6, C to G), thereby confirming that this separation is not due to RT differences.

Fig. 4 Task-specific representation of choice.

(A) Example MFC choice cells, split by choice (yes or no) and task. (B) Population choice decoding accuracy was significantly greater in MFC than in HA (across all trials, Δtrue = 19% versus empirical null, P < 1 × 10−3). (C) MFC cells represent choice and not the ground truth (i.e., new or old; memory trials only). (D) Population summary (neural state space) of choice-related activity in MFC, plotted in 3D space derived using MDS. (Top) Variability due to response modality. The highlighted planes connect the points of state space occupied by activity when using button presses (purple) or saccades (green). (Bottom) Variability due to task type. The highlighted planes connect the points of state space occupied by activity in the same task. (E) Choice-decoders trained in one task do not generalize to the other task (bin size: 500 ms, step size: 16 ms). (F) Same as (E), but for a fixed 1-s time window starting at 0.2 s after stimulus onset. (G) Choice decoding generalizes across effectors [see (D)]. (H) Generalization index of choice decoding (see Materials and methods) to summarize (F) and (G). The representation of choices generalized across response modality but not task. (I) Generalization between different subtasks of the categorization task but not between task types. The colored bars indicate the 5th to 95th percentile of the null distribution. (J) (Left) State-space trajectories for the four conditions arising from the combination of response (yes or no) and task (categorization or memory). (Right) Trajectory similarity, computed in an 8D latent space (recovered using GPFA, see Materials and methods) across the eight conditions arising from the combinations of choice, effector type, and task. (K) Decoder weight of each cell for decoding choice during the categorization and memory task. The cells in the top 25th percentile are shown in black. The inset shows the angle created by the vector [ωicat,ωimem] with respect to the x axis of the cells marked in black.

We used multidimensional scaling to visualize the population activity for the eight combinations of choices, task types, and response modality (Fig. 4D, and see Materials and methods). The resulting geometrical configuration indicates that choice decoding generalizes across response modality (Fig. 4D, top) but not across task types (Fig. 4D, bottom). We therefore computed the cross-task generalization performance of a decoder trained on choices during one task and tested on the other. We performed this analysis across time (Fig. 4E; see fig. S7B for this analysis shown separately for pre-SMA and dACC) and also in a single poststimulus time bin (Fig. 4F). To avoid confounds due to response time differences, we performed the fixed window analysis (Fig. 4F) only for the control task, where the timing between tasks was identical (fig. S6B). While the choice signal did not generalize across task types (Fig. 4F), it did generalize across response modality within the same task type (Fig. 4G). Quantifying this observation with the generalization index confirmed this impression (Fig. 4H). We also examined choice signals in the HA. Although choice signals were comparatively weak in the HA (fig. S7E and Fig. 4B; see Fig. 4B for statistics), they nevertheless exhibited a pattern of generalization similar to that seen in the MFC (fig. S7, E and F).

To test the possibility that any task might yield a distinctive choice axis that does not generalize to any other task, we considered the four subtasks that make up the categorization trials in a given session (the target category can be any one of the four possible image categories). We tested whether the choice signal generalizes across these subtasks by training and testing across blocks requiring different categorizations. Choice decoding generalized across all subtasks in the categorization task but not the memory task (Fig. 4I). Next, we compared the dynamics of the population activity between the eight conditions arising from the combination of choice, effector type, and task in a state-space model recovered using Gaussian process factor analysis (GPFA) [(45), and see Materials and methods]. Comparing the pairwise similarity between the trajectories in state space (Fig. 4J, left) within the first 500 ms after the stimulus onset revealed that the patterns of dynamics in state space first cluster by task type (Fig. 4J, right; see movie S1).

We next examined whether the population-level analysis relied on different sets of neurons to decode choice in each of the two tasks. We determined how individual cells are recruited by a linear decoder (44, 46). For each cell, we quantified its importance (46) for both the memory and categorization choice decoder (Fig. 4K). We then plotted the degree of specialization for each cell on the basis of its importance in each task (see Materials and methods). Cells that report choice independently of task should lie on the diagonal (i.e., an angle of π/4). Instead, the distribution of angles was significantly bimodal across all cells (Fig. 4K, inset plot; P < 1 × 10−5, Hartigan dip test), with modes centered away from the diagonal. Despite this bimodality, we could still use the cells that are the most “useful” in one task to train a new decoder that can predict choice well above chance (although significantly weaker compared with using all cells) in the other task (fig. S7C). This is not an example of cross-task generalization, because we are fitting a new decoder.

Task-dependent spike-field coherence between MFC cells and HA LFP

It is thought that the selective routing of decision-related information between the HA and MFC is coordinated by interareal spike-field coherence (24). We therefore asked whether MFC neurons phase-lock to ongoing oscillations in the LFP in the HA and, if so, whether the strength of such interactions is modulated by task demands. We performed this analysis for the 13 subjects and 33 sessions for which we simultaneously recorded from both areas (Fig. 5A). In the following, we only used neural activity from the 1-s baseline period that precedes stimulus onset, to avoid confounds related to stimulus onset–evoked activity. Individual cells in the MFC showed strong task modulation of MFC to HA spike-field coherence (Fig. 5B shows a single-cell example in the dACC). At the population level, MFC cells showed significantly stronger theta-band coherence with HA oscillations during the memory task than during the categorization task (Fig. 5C, 8822 cell electrode pairs; P < 1.3 × 10−7, paired t test, measured at 5.5 Hz; see fig. S9, B and C, for additional controls). This was the case for both MFC-hippocampus and MFC-amygdala interactions (Fig. 5D, n = 3939 and 4884 and P = 8.8 × 10−4 and 4.3 × 10−5, respectively, paired t test). This effect was due to changes in phase preference, as there was no significant difference in HA LFP power between the tasks (Fig. 5E, P = 0.08, paired t test of signal power at 5.5 Hz, estimate across all 8822 cell-electrode pairs). Of the 767 MFC cells, a significant subset of ~100 cells were phase-locked to the theta-band HA LFP (fig. S9A), with the largest proportion preferring 3 to 8 Hz.

Fig. 5 Modulation of interareal spike-field coherence by task demands.

(A) Analysis approach. Inset shows that only data from the baseline was used [except in panel (H)]. (B) Spike-field coherence for a cell in dACC relative to all channels in the ipsilateral hippocampus. (C) Phase-locking of MFC cells to HA LFP. (Top) Average interarea PPC of all cell-electrode pairs for each task. (Bottom) Significance of difference between tasks; peak difference was at f = 5.5 Hz. Dashed line shows the threshold (P = 0.05/56, Bonferroni corrected). (D) Difference in average interarea PPC at f = 5.5 Hz between task conditions for all possible cell-electrode pairs (from left to right, n = 8822 electrode pairs, P = 1.3 × 10−7; n = 3938, P = 8.8 × 10−4; n = 4884, P = 4.3 × 10−5; paired t test). (E) Average spike-triggered power was not significantly different between the two tasks (paired t test, n = 8822 cell electrode pairs, P = 0.08). (F) Single-neuron analysis of choice cells. Importance index assigned by the decoder to each cell for decoding choices in either task. Selected choice cells are indicated in color. (G) MFC-HA spike-field coherence for choice cells. (Top) Average PPC for all choice cells in MFC (209 cells, 2384 cell-electrode pairs) for the two tasks. (Bottom) Significance of difference between tasks, shown separately for memory and categorization choice cells (n = 1176 and 906 cell-electrode pairs, respectively). Only memory choice cells show a significant difference in the gamma band (P = 2 × 10−6, t test). (H) Difference in spike-field coherence between true-positive (correct retrieval) and false negative (incorrect retrieval) trials measured in the [0.2 s, 1.2 s] window after the stimulus onset, shown separately for memory choice cells (left panel) and categorization choice cells (right panel) in the theta-frequency (4 to 10 Hz) and gamma-frequency (30 to 80 Hz) bands. PPC was significantly stronger in correctly retrieved trials in the theta band for memory choice cells (Δmem = 0.003, P = 0.002; Δcat = −0.001, P = 0.3; paired t test) and in the gamma band for both types of choice cells (Δmem = 0.002, P = 0.005; Δcat = 0.004, P = 7.2 × 10−8; paired t test). (I) Spike times of HA cells are coherent with local theta-band (3 to 8 Hz) LFP. (J) Average local PPC in the HA did not differ significantly as a function of the task (f = 5.5 Hz; P = 0.51, paired t test, n = 2321 cell-electrode pairs). Error bars in panels (D) and (H) denote the 95% confidence interval (bootstrap, n = 10,000 iterations). All other error bars are SEM.

To determine whether there is a relationship between the tuning of cells in MFC and their interarea coherence with HA, we selected for choice cells independently in the categorization and memory task (see Materials and methods for selection model; note selection controls for RT differences). This approach revealed that 101/767 and 82/767 cells were significantly modulated by choice during the memory and categorization task, respectively (P < 0.001 versus chance for both; see Fig. 4A, cells 2 and 1, respectively, as an example). Single-neuron decoding showed that it was not possible to decode the subject’s choice in the categorization task from choice cells selected in the memory task and vice versa (fig. S5, A to D). Single-neuron analysis revealed that cells preferring either “no” or “yes” choices were present in approximately equal proportions in both tasks (fig. S5B). The removal of the selected choice cells from a population decoding analysis with access to all recorded neurons significantly diminished decoding performance (fig. S5, F and G). Notably, each of the selected cells had a high importance index, as determined from population decoding (Fig. 5F). Considering only the MFC choice cells revealed that this subset of cells similarly increased their phase-locking during the memory task (Fig. 5G, top), with the strongest effect, again, in the theta range [peak frequency (fpeak) = 5 Hz, P = 1 × 10−6, paired t test]. Both categorization and memory choice cells showed this pattern of modulation (Fig. 5G, bottom). The memory choice cells exhibited an increase in gamma-band coherence (Fig. 5G, fpeak = 38.5 Hz, P = 2 × 10−6, paired t test). The extent of phase-locking of choice cells following stimulus onset (0.2 to 1.2 s) during the memory task was significantly stronger for correctly retrieved trials than for forgotten old trials, indicating behavioral relevance for memory retrieval (Fig. 5H). Lastly, to exclude the possibility that this interarea effect was due to task-dependent changes within the HA, we examined the phase-locking properties of HA cells to their own locally recorded LFP (LFP and spiking activity is recorded on separate electrodes; see Materials and methods). The spiking activity of 331/663 HA cells was significantly related to the theta-frequency band LFP (Fig. 5I, shown for f = 5.5 Hz). The strength of this local spike-field coherence was, however, not significantly different between the two tasks (Fig. 5J, P = 0.61, paired t test, n = 2321 cell-electrode pairs).

Discussion

We investigated the nature of flexible decision-making in the human brain by probing how the strength and/or geometry (44) of neural representations of stimulus memory, stimulus category, and choice is modified when subjects switch between a memory and a categorization task. We found evidence for both kinds of neural representation changes resulting from changing task demands for a subset of the studied variables. In the MFC, both the strength and geometry of representation of stimulus memory changed as a function of task demands. In contrast, in the HA, both the strength and geometry of the representation of stimulus memory were insensitive to task demands (Fig. 3, D and H). Our finding of memory signals in the amygdala supports the hypothesis (47, 48) that the amygdala contributes to recognition memory by signaling stimulus familiarity. Representation strength of stimulus category in both the HA and MFC was stronger in the memory task, but the geometry of this representation was also modulated by the task in the MFC (Fig. 3G, right). Overall, these results show that the geometry of the representations (as assessed by cross-task generalization) of stimulus familiarity and memory were significantly less sensitive to task demands in the HA compared with the MFC (Fig. 3, G and H).

At the population level in the MFC, choices in both the memory and categorization task were decodable with high reliability, but these decoders did not generalize across the two tasks. Choice decoding generalized across subtasks of the categorization task and across changes of response modality in both tasks, indicating that changes in representations were due to switching between a task that requires memory retrieval and one that does not. While the choice signal was significantly weaker in the HA, this same pattern of generalization also held for the HA, suggesting the possibility that the task demand–dependent choice representation we found in the MFC is widely distributed in the brain. A group of task demand–dependent cells in the MFC were choice cells, which preferentially signal behavioral decisions for either memory or categorization decisions irrespective of response modality and regardless of the ground truth of the decision. Thus, from the point of view of downstream areas, neurons formed two separate decision axes: one for memory-based decisions and one for categorization-based decisions. These two decision axes were instantiated selectively so that they were only present when required by the current task.

These findings contrast with prior work on task switching involving different tasks that required purely perceptual decisions, which found a single decision axis in the monkey prefrontal cortex, with task-irrelevant attributes also represented (3). We found that memory-based choices add a second decision axis, which is present only when decisions engage memory retrieval processes. Although task-sensitive representations of choice have been shown in recordings from rodents and nonhuman primates during perceptual decision-making (2, 3, 49, 50) and in human neuroimaging (51), our data reveal choice representations that specifically signal recognition memory–based choices at the single-cell level. It has long been appreciated that the frontal lobes are critical for initiating and controlling memory retrieval (30, 5254). Neuroimaging reveals that patterns of activity in some frontal and parietal areas are modulated by memory retrieval demands (34, 35, 37, 38, 55), whereas memory-related activity patterns in the MTL are comparatively insensitive to retrieval demands (34). These findings have led to the proposal that the memory retrieval network consists of specialized processes separate from those used for other kinds of decisions (6, 56, 57). The memory-choice axis we describe is a potential cellular substrate for this critical aspect of human cognitive flexibility. Future work is needed to investigate whether similar principles also apply to hippocampus-dependent associate or source memory–based (5860) decisions, which we did not assess here (we probed recognition memory).

A second group of cells that we characterized in the MFC signal the currently relevant goal (task type and response modality) throughout the task. These cells switched their activity pattern when instructions indicated a change in task demands. Although these switches were rapid, they were not instantaneous, likely reflecting the cost of switching between memory retrieval and categorization modes (6163). We hypothesize that these cells facilitate the holding of the active task set in working memory (64, 65) and configure brain networks in preparation for appropriate execution of the instructed task (12, 34, 66, 67). Task-switching costs are a much investigated aspect of cognitive flexibility (39, 6163), but how they arise and why some task switches are more difficult than others remain poorly understood. The MFC cells we describe here offer an avenue to directly investigate these questions.

Finally, we uncovered a possible mechanism by which memory-based information can be routed dynamically between the MFC and HA when a task requires memory retrieval. Changing long-range synchronization of neural activity is thought to be a way by which functional connectivity between brain areas can be changed flexibly (6871). Here, we reveal a specific instance of this phenomenon at the cellular level in humans in the form of changes in the strength of cortico-hippocampal and cortico-amygdala functional connectivity. Hippocampus–medial prefrontal cortex (mPFC) functional connectivity in rodents supports spatial working memory (24) and is prominent during both navigation and rest (7274). Similarly, amygdala-mPFC functional connectivity supports flexibly switching between aversive and neutral behaviors depending on learned cues (75). But it is not known whether these pathways serve a role in long-term memory retrieval in humans and, if so, whether this retrieval can be engaged selectively. Similarly, in humans, MTL-PFC connectivity changes, as measured by fMRI, have been related to control demands over memory retrieval (36, 76), but it remains unclear how these indirect metrics relate to long-range synchronization as measured in rodents. Here, we show that MFC-HA connectivity is selectively enhanced during the memory task, indicating that patterns of interareal connectivity change in preparation of initiating memory retrieval (77, 78). The extent of synchrony after stimulus onset is stronger when a memory is successfully retrieved compared with when it is forgotten. Memory choice cells in the MFC exhibited enhanced gamma-frequency band coordination of their spiking activity with the HA LFP, and this modulation was behaviorally relevant after stimulus onset, which reveals a specific cellular-level instance of a role for gamma oscillation–mediated coordination of activity between distant brain regions (24, 79) in human memory retrieval.

Materials and methods

Subjects

Subjects were 13 adult patients being evaluated for surgical treatment of drug-resistant epilepsy that provided informed consent and volunteered for this study (see table S1). The institutional review boards of Cedars-Sinai Medical Center and the California Institute of Technology approved all protocols. We excluded potential subjects who did not have at least one depth electrode in medial frontal cortex.

Electrophysiology

We recorded bilaterally from the amygdala, hippocampus, dACC, and pre-SMA using microwires embedded in hybrid depth electrodes (81). From each microwire, we recorded the broadband 0.1 to 9000 Hz continuous extracellular signals with a sampling rate of 32 kHz (ATLAS system, Neuralynx Inc.). Subjects from which at least one well-identified single neuron could not be recorded were excluded.

Spike sorting and single-neuron analysis

The raw signal was filtered with a zero-phase lag filter in the 300 to 3000 Hz band, and spikes were detected and sorted using a semiautomated template-matching algorithm (82, 83). All PSTH diagrams were computed using a 500-ms window with a step size of 7.8 ms. No smoothing was applied.

Electrode localization (relevant for Fig. 1)

Electrode localization was performed based on postoperative MRI scans. These scans were registered to preoperative MRI scans using Freesurfer’s mri_robust_register (84) to allow accurate and subject-specific localization. To summarize electrode positions and to provide across-study comparability, we also aligned the preoperative scan to the MNI152-aligned CIT168 template brain (85) using a concatenation of an affine transformation followed by a symmetric image normalization (SyN) diffeomorphic transform (86). This procedure provided the MNI coordinates that are reported here for every recording location. The electrode locations shown on the Atlas Brain (Fig. 1, C and D) are for illustration only. Apparent localization outside the target area or in white matter is due to usage of an Atlas Brain alone.

Eye tracking (relevant for fig. S1)

Gaze position was monitored using an infrared-based eye tracker with a 500-Hz sampling rate (EyeLink 1000, SR Research) (87). Calibration was performed using the built-in nine-point calibration grid and was only used if validation resulted in a measurement error of <1 degree of visual angle (dva) (average validation error was 0.7 dva). We used the default values for the thresholds in the Eyelink system that determine fixation and saccade (eye movement) onsets.

Task

Each session consisted of eight blocks of 40 trials shown in randomized order. At the beginning of each block, an instruction screen told subjects verbally the task to be performed for the following 40 trials (categorization or recognition memory), the response modality to use (button presses or eye movements), and which visual category is the target (for categorization task only; either human faces, monkey faces, fruits, or cars; order was pseudorandom so that each image type was selected as the target at least once) (see Fig. 1). The task to solve was either “Have you seen this image before, yes or no?” or “Does this image belong to the target category, yes or no.” Odd-numbered blocks (1, 3, 5, and 7) were categorization blocks; even numbered blocks were memory blocks (2, 4, 6, and 8). Button presses (yes or no) were recorded using a response box (RB-844, Cedrus Inc.). Eye movements to the left or right of the image served as responses in the eye movement modality (left = yes, right = no). The mapping between button and screen side and yes/no responses was fixed and did not change; “yes” was on the left and “no” was on the right. Subjects were reminded that left = yes, and right = no, at the beginning of each of the eight blocks. In the first block, all images were novel (40 unique images). In all subsequent blocks, 20 new images were shown randomly intermixed with 20 repeated images (the “old set”). The 20 repeated images remained the same throughout a session. We used entirely nonoverlapping image sets for patients that completed multiple sessions. The response modality (button presses or eye movements) was initially selected randomly and then was switched in the middle of each block (an instruction screen in the middle of each block showed the response modality to be used for the remainder of the block). In sessions where eye tracking was not possible because of problems with calibration (five sessions in three patients; see table S1), all trials used the button presses as the response modality. No trial-by-trial feedback was given. In between image presentations, subjects were instructed to look at the fixation cross in the center of the screen.

Control task (relevant for fig. S6)

In 5 of the 13 subjects in this dataset (6/33 sessions), we ran an additional control task in order to help determine whether neural responses reflected processing of stimuli, of decision variables, or of motor response plans. Unlike the standard task where the subjects could respond at any time after the stimulus onset (thus making it difficult to distinguish decision from choice), in this control task the subjects were instructed to wait until the response cue in order to register their answer, either with a button press or with a saccade. The stimulus was presented for a fixed amount of time (1-s duration) and after a 0.5-to-1.5-s delay period, the subjects were asked to respond to the question relevant for that block.

Mixed-effects modeling of behavior (relevant for Fig. 1 and fig. S1)

For the group analysis of behavior, we used mixed-effects models of the form y = Xβ + Zb + ε, where y is the response, X is the fixed-effects design matrix, β is the fixed-effects coefficients, Z is the random-effects design matrix, b is the random-effects coefficient, and ε is the error vector. In all analyses, we used a random intercept model with a fixed slope. The grouping variable for the random effects was the session ID. The reported P values in the main text correspond to the fixed-intercept for the relevant variable. In the case of measuring the effect of number of expositions (i.e., number of times an image was seen) on the subject’s accuracy during the memory trials, we used a mixed-effects logistic regression with the independent variable as an ordinal-valued whole number ranging from 1 to 7. The response was a logical value indicating success or failure on each memory question. Prior to running any analysis of reaction time data, we excluded outliers from the distribution using the following procedure: A sample was considered an outlier if it was outside the 99th percentile of the empirical distribution.

Reaction time matching procedure

As a control, we matched for RTs between the two tasks (categorization and memory) to exclude for potential differences due to difficulty. To achieve this, we first added noise to all reaction times (SD = 1 ms), followed by locating pairs of trials with RTs that were equal to within a tolerance of 0.1 s. Matching pairs were then removed, and this procedure was repeated iteratively until no further matches could be found. Unmatched trials were excluded (resulting in reduced statistical power owing to fewer available trials). We only used the resulting match if the RTs between the two groups were not significantly different. If not, the procedure above was repeated.

Selection of visually (VS) and memory-selective (MS) cells (relevant for fig. S3)

A cell was considered a VS cell if its response covaried significantly with visual category, as assessed using a 1×4 analysis of variance (ANOVA) test at P < 0.05. For each selected cell, the preferred image category was set to be the image category for which the mean firing rate of the cell was the greatest. All trials were used for this analysis. MS cells were selected using the following linear modelfrcell1+β1·category+β2·new/old+β3·rt where “category” is a categorical (1×4) variable, “new/old” is a binary variable, and “rt” is a continuously valued variable. A cell was determined to be memory-selective if the t-statistic for β2 was significant with P < 0.05. We excluded the first block of trials (40 images) from the analysis, in order to keep the number of new and old stimuli the same. Spikes were counted for every trial in a 1-s window starting at 200 ms after stimulus onset.

Selection of choice cells (relevant for fig. S5)

Choice cells were selected using a regression model applied to the firing rate in a 1-s-size window starting 200 ms after stimulus onset. We fit the following regression modelfrcell1+β1·category+β2·response+β3·rt where the response is binary (yes or no), category is a categorical variable with four levels, and RT is the reaction time. We fit this model separately to trials in the memory and categorization condition, assuring independent selection of cells. RT was included as a nuisance regressor to control for reaction time differences between the two possible responses (see fig. S1A). A cell qualified as a choice cell if the t-statistic of the β2 term was significant at P < 0.05 for at least one of the two task conditions. The response preference of significant cells for either yes or no was determined based on the sign of β2 (positive = yes, negative = no). Notice that the selection process uses separate trials for memory choice cells and categorization choice cells. All trials, regardless of whether the answer was correct or incorrect, were used for selection. To estimate the significance of the number of selected cells, we generated a null distribution by repeating the above selection process 1000 times after randomly reshuffling the response label. We estimated this null distribution separately for choice cells in the memory and categorization condition and used each to estimate the significance of the number of selected cells of each type.

Chance levels for cell selection (relevant for figs. S2, S3, and S5)

To estimate the chance levels for cell selection, we repeated above procedures for selection of visual category, memory selective, and choice cells after randomly scrambling the order of the labels determining the category membership being selected for (yes/no response, visual category, and new/old ground truth, respectively). We repeated this procedure 1000 times.

Single-cell decoding (relevant for fig. S5)

Single-cell decoding was done using a Poisson naïve Bayes decoder. The features used were spike counts in a 1-s window, in the interval [0.2 s, 1.2 s] relative to stimulus onset. The decoder returns the probability of a class label, given the observed spike count. The class label was binary (“yes” or “no”). The model assumes that the spike count is generated by a univariate Poisson distribution, and a separate mean rate parameter (λ) is fit to each feature-class pair. For a new observation, class membership is determined on the basis of the likelihood value. Notice that we used a single spike count as a feature, so the naïve assumption of the decoder is no longer relevant in this case.

Population decoding (relevant for Figs. 2, 3, and 4 and figs. S2 to S5, S7, and S8)

Single-trial population decoding was performed on a pseudo-population assembled across sessions (88). We present decoding results for a variety of task variables: (i) image category, (ii) new versus old, (iii) choice during memory trials, (iv) choice during categorization, (v) task, and (vi) response type. In order to estimate the variance of the decoding performance, on each iteration of the decoder (minimum of 250 iterations), we randomly selected 75% of the cell population that was being analyzed. For example, to measure choice decoding in MFC (as shown in Fig. 4), we would randomly select 575/767 cells on each iteration of the decoder. The total number of available cells depended on the variable that was being decoded. For example, for response type decoding, the number of cells in MFC was 593, because 28/33 sessions included both response types. We matched the number of trials per condition contributed by each cell that was selected to participate in the population decoding. For most task variables (image category, new/old, context, effector-type), the number of samples from each cell was equal, because the task structure remained the same across all subjects and sessions. For choice decoding, however, the number of instances varied, because the subjects were free to respond with a “yes” or a “no” for each stimulus. We therefore matched the numbers for the smallest group across all subjects. This matching procedure can further reduce the number of cells we included in the decoding that do not have the minimum number of trials needed per condition. For the population decoding and cross-condition generalization of familiarity presented in Fig. 3, we used all image categories for which the subjects showed above-chance recognition performance (see fig. S1B). Therefore we used images of cars, fruits, and faces, excluding images of monkeys from the analysis. For all other analysis, we used all available categories.

A series of preprocessing steps were carried out before training the decoder. Firing rates for each cell were first de-trended (to account for any drift in the baseline-firing rate) and then normalized (z-scored) using the mean and standard deviation estimated from the training set. We then performed 10-fold cross validation using a linear support vector machine (SVM) decoder to estimate performance, as implemented by the “fitcecoc” function in MATLAB. We used an SVM with a linear kernel and a scale of 1. Decoding results are reported either as a function of time or in a fixed time window. Time-resolved decoding was done on spike counts measured in a 500-ms moving window, with a 16-ms step size. For fixed-window decoding, we used spike counts in a 1-s window. The location of the window depended on the analysis. In Fig. 2, for example, we used a [−1, 0] window relative to stimulus onset for task type and response type decoding. In Fig. 3, we used spike counts in a [0.2, 1.2] window relative to stimulus onset for decoding image category and new/old.

Null models for testing significance of decoding performance

Throughout the manuscript, we compare the performance of our decoders against the 95th percentile of a null distribution. The way that this null distribution is generated depends on the variable being decoded. For variables such as image category, new versus old, and response (i.e., yes versus no), we used a simple shuffling procedure for the labels. For variables such as task type, which had structure over time (memory blocks were always preceded by categorization block), small drifts in firing rate might lead to inflated decoding accuracy. Therefore, for such variables, the shuffling was done in such a way as to preserve their temporal relationship. Specifically, we offset (i.e., circular shift) the labels by a random integer value (sampled from the range ±10 to 20 trials). In the case of task decoding from the baseline firing rate, this is a very conservative measure of the null decoding performance, because many trials retain their original label, thereby inflating the accuracy. This also means that the mean performance of the null distribution will not be the theoretical chancel level. In the case of task decoding, the theoretical chance level is 50% (binary classification). Using the circular shift method for scrambling labels, the mean of the null distribution was ~60%.

To compare the performance between different decoders, for example choice decoding from the HA versus MFC population, we constructed an empirical null distribution from the pairwise differences in the performance of these two decoders trained using the shuffled labels. For example, if we get N estimates of the null performance (i.e., after shuffling the labels) of the HA decoder and N estimates of the null performance of the MFC decoder, we construct a distribution of the N•N = N2 pairwise differences. We can then compute the significance of the true difference in decoding performance between MFC and HA, Δtrue, relative to this distribution. The variance of the null distribution is sensitive to the number of trials available for decoding because it changes the resolution (step size) by which decoding accuracy can change. For example, for 10 trials, the accuracy can take values from 0 to 1 in increments of 0.1. This results in different values for the 95th percentile of the null distribution and is the reason why in some cases a given difference in decoding accuracy is significant while it is not in others. Unless otherwise specified, all P values for comparing decoding performance between conditions or brain areas are calculated using this approach. In the one case where the number of trials in a condition was too low to reliably estimate the null distribution (fig. S5I) and for comparing the generalization index (fig. 3J) we used a bootstrap test for equality of means (89) to compare the two conditions to assign a P value to the true difference (repeated 1000 times to estimate the null distribution).

Multidimensional scaling (relevant for figs. 3 and 4)

Multidimensional scaling (MDS) was used only for visualization. We computed MDS using Euclidean distances (MATLAB function “mdscale’) on z-scored spike count data in the [0.2 s, 1.2 s] window relative to image onset. In Fig. 3E, for example, MDS was computed on the activity across the entire population of HA and MFC cells, averaged across the eight conditions plotted (new/old ⊗ task ⊗ image category, where ⊗ denotes the Cartesian product). Here the image category was restricted to images of human faces and fruits, for visualization purposes. For the cross-condition generalization performance, we use all four image categories. In Fig. 4D we compute MDS on the population of MFC cells, averaged across eight conditions (response ⊗ task ⊗ effector, where ⊗ denotes the Cartesian product). In all cases, we use MDS to map the neural activity to a 3D space.

Normalized weight metric (relevant for Figs. 4 and 5 and figs. S5, S7, S8, and S10)

The normalized weight metric is computed from the weight that a decoder assigns to a particular cell for a given classification. This weight is denoted as wit, where the index i denotes the cell, and the index t denotes the condition (for example, categorization or memory). The weight is converted into a normalized measure called an importance index, defined as

ωit=|wit|1ni|wit|

State-space analysis (relevant for Fig. 4I)

We used Gaussian-process factor analysis (GPFA) (45) to analyze the dynamics of the average population activity for the eight conditions arising from the combination of choice (yes, no), response modality (button press, saccade), and task (memory, categorization). The recovered latent space was eight-dimensional (8D), and all similarity measurements between trajectories were performed in this space (not in the 3D projections shown in the figure). The activity was binned using 20-ms windows. All analysis was computed and visualized using the DataHigh (90) MATLAB toolbox. Similarity measurements between two conditions were computed and averaged over the first 500ms after stimulus onset as followssim(r1(t),r2(t))=r1(t)r1(t)·r2(t)r2(t) where r1(t) and r2(t) are the 8D state-space trajectories for conditions 1 and 2 [ r1(t) and r2(t) indicate the derivatives over time], respectively.

ANOVA model (relevant for figs. S4 and S11)

We used a single-cell ANOVA model to tease apart the contributions of choice, visual category, memory, and response time on the firing rate of a cell. The model was of the following formfrcellβ1·category+β2·familiarity+β3·choice+β4·rtwhere frcell is the mean firing rate in a fixed window (0.2 to 1.2 s following stimulus onset) or a moving window of 500 ms to analyze the time course. The ANOVA model is fit independently at each point of time. We then compute the F-statistic for each of the regressors and report the average F-statistic across the entire population of recorded cells, fit twice to each cell for the memory and categorization task (figs. S4, D and E, fig. S11). To compare the effects of task on the representation of individual variables, we compare the distribution of F-statistics estimated separately on each task, for each cell in the population. We use this approach as a measure of modulation in the strength of representation for a variable induced by task switching. This comparison does not make predictions about generalizability from one task to the next, because the model is fit independently.

Generalization index (relevant for Figs. 3 and 4)

To compare the within-condition decoding to the across condition generalization, we used a generalization index defined as followingg=crosschancewithinchance where “within” is the decoding performance within condition, “cross” is the decoding across condition, and “chance” is the chance decoding performance for the variable of interest (choice = 0.5, new/old = 0.5, familiarity = 0.5, image category = 0.25).

Spike-field coherence analysis (relevant for Fig. 5 and fig. S9)

LFP preprocessing

The local-field potential recordings were high-pass filtered at 1 Hz. The raw recordings, sampled at 32 kHz, were then downsampled to 500 Hz. The downsampling procedure was done with the “resample” command in MATLAB, which applies the appropriate antialiasing filter prior to reducing the sampling rate. For each session, we screened all MFC and HA electrodes in order to make sure that there were no artifacts that could contaminate the spike-field metrics. We excluded all electrodes with interictal discharges (IEDs) visible in the raw trace (by visual inspection). Specifically, in screening for IEDs, we looked for large stereotyped, recurring transients in the raw recording that did not correspond to cellular spiking activity. The presence of such transients would disqualify an electrode from further consideration.

Spike-field coherence (SFC)

All spike-field coherence analysis was performed on snippets of the LFP extracted around the spike. We extract snippets for every cell-electrode pair. For example, to measure interarea SFC between a single cell in pre-SMA and HA LFPs, we extracted n snippets each (n = number of spikes) from each of the eight ipsilateral electrodes in hippocampus and eight electrodes in the ipsilateral amygdala. For sessions where we used a local reference (i.e., bipolar referencing), we exclude the reference wire. For intra-area coherence (for example, HA spikes to HA field) we also exclude the wire on which the cell was recorded to avoid contamination by spike waveform. For each snippet and for each cell-electrode pair, we compute the spike-triggered spectrum using the FieldTrip “mtmconvol” method, which computes the Fourier spectrum of the LFP around the spikes using convolution of the complete LFP traces. The spectrum was computed with a single “hanning” taper, at 56 logarithmically spaced frequencies ranging from 2 Hz on the low end to 125 Hz on the high end. The length of the snippet window was dynamic as a function of the frequency examined; the snipped length was set to equal to two cycles of the underlying frequency at which the spectrum was estimated (i.e., 2 Hz→2 s snippet). We estimated the phase for each snippet and for each of the 56 frequencies from the complex-valued Fourier coefficients (i.e., phasor). We used the pairwise phase consistency (PPC) metric as the measure of coherence. For the spike-triggered power, we compute the magnitude of the spectral coefficients returned by the Fourier transform (also computed for each cell-electrode pair) for each snippet and averaged the spectra. Unless otherwise stated, all SFC results in the paper are based on spikes recorded during the baseline period between trials (1-s window preceding stimulus onset).

Group comparisons using the SFC metric

When comparing two or more groups using PPC (such as memory versus categorization), we balanced the number of spikes between the two groups. To reduce bias involved in subsampling the larger group, we resampled the spikes from the two groups 200 times and computed the PPC metric on each iteration. The final coherence measure for a given cell-electrode pair was an average across all 200 iterations.

To ensure that the underlying local field potential does not vary in a consistent way across conditions, we compare the distribution of average voltage values for each of the conditions in our spike-field coherence analysis. In the case of the task contrast during baseline (i.e., memory versus categorization), we show the distribution of area under the curve (AUC) values computed separately for each electrode in the amygdala and hippocampus (fig. S9D shows that there was no significant difference). The AUC for each electrode is computed using the average baseline magnitude across memory and categorization trials. In the case of the spike-field coherence results during the stimulus onset (Fig. 5H), to reduce any potential confounds related to event-related potentials, we used only sessions with local referencing (bipolar). The local reference (set to one of the eight microwires in the electrode cluster implanted in each brain area) significantly diminishes the magnitude of any event-related potentials after stimulus onset. To confirm this, we repeated the AUC analysis mentioned above, for the contrast in Fig. 5H [i.e., true positive (TP) versus false negative (FN)]. The results (shown in fig. S9E) show that there is no significant difference between the two conditions of interest.

Supplementary Materials

science.sciencemag.org/content/368/6498/eaba3313/suppl/DC1

Supplementary Text

Figs. S1 to S11

Table S1

Movie S1

References and Notes

Acknowledgments: We thank the members of the Adolphs and Rutishauser labs for discussion, Columbia Theory Center members F. Stefanini and M. Rigotti for sharing their population decoding analysis expertise, and the staff and J. M. Chung and C. M. Reed of the Cedars-Sinai Epilepsy Monitoring unit for their support. We thank all subjects and their families for their participation. Funding: This work was supported by NIMH (R01MH110831 to U.R.), the BRAIN initiative through the NIH Office of the Director (U01NS103792 to U.R.), the Caltech NIMH Conte Center (P50MH094258 to R.A. and U.R.), the National Science Foundation (CAREER Award BCS-1554105 to U.R. and NeuroNex Program award DBI-1707398 to S.F.), a Memory and Cognitive Disorders Award from the McKnight Foundation for Neuroscience (to U.R.), and the Simons Foundation Collaboration on the Global Brain (542941 to R.A. and PG007079 to S.F.). Author contributions: J.M., U.R., and R.A. designed the study. J.M. performed the experiments. J.M., S.F., and U.R. analyzed the data. J.M., U.R., R.A., and S.F. wrote the paper. A.N.M. performed surgery and supervised clinical work. Competing interests: None. Data and materials availability: Data needed to reproduce results (firing rates of all neurons versus time, precomputed phase-coherence measures for all neurons) have been deposited at OSF (80).

Stay Connected to Science

Subjects

Navigate This Article