Research Article

Hippocampal sharp-wave ripples linked to visual episodic recollection in humans

See allHide authors and affiliations

Science  16 Aug 2019:
Vol. 365, Issue 6454, eaax1030
DOI: 10.1126/science.aax1030

Sharp-wave ripples in the hippocampus

What are the brain mechanisms responsible for episodic memory retrieval? Norman et al. investigated epilepsy patients who had electrodes implanted in the hippocampus and a variety of cortical areas. Using a visual learning paradigm, they examined the temporal relationship between the incidence of hippocampal sharp-wave ripples and recall. Effective encoding of visual information was associated with higher incidence of ripples. Successful recall was preceded by an increased probability of ripples, which were also associated with transient reemergence of activation patterns in higher visual cortical areas. Hippocampal ripples may thus boost recollections during episodic memory retrieval.

Science, this issue p. eaax1030

Structured Abstract

INTRODUCTION

Sharp-wave ripples (SWRs) are rapid bursts of synchronized neuronal activity elicited by the hippocampus. Extensive study of SWRs, mainly in the rodent brain, has linked these bursts to navigation, memory formation, and offline memory consolidation. However, fundamental questions remain regarding the functional meaning of this striking example of network synchrony. Perhaps the most glaring unknown is the relationship between SWRs and conscious cognition. We still do not know what cognitive process, if any, is linked to the emergence of SWRs; to put it simply, we still do not know what an animal thinks about (if anything) when the hippocampus elicits a ripple. Furthermore, the potential role of SWRs in human episodic memory is still largely unknown. Thus, studying this phenomenon in conscious, awake human patients opens a unique window, as it allows direct examination of detailed verbal reports with respect to SWR occurrences.

RATIONALE

We took advantage of the unique ability of humans to communicate verbally about their inner cognitive state to examine the role of SWRs in memory formation and retrieval, using intracranial electrophysiological recordings in patients. This approach allowed us to study free recall, the process of self-initiated, internal generation of memories. It is a uniquely powerful approach because it isolates the process of recall from external stimulation.

RESULTS

Our study revealed three major aspects linking SWRs to human declarative memory. First, the SWR rate during picture viewing (i.e., memory encoding) predicted subjects’ subsequent free-recall performance. Second, a transient increase in SWR rate preceded the verbal report of recall by 1 to 2 s. This increase was content-selective, recapitulating the same picture preferences observed during viewing. Finally, during recollection, high-order visual areas showed content-selective reactivation coupled to SWR emission.

CONCLUSION

By direct recordings of electrophysiological events in the brains of individuals who could inform, in real time, on their cognitive state, we were able to demonstrate and characterize an important role of SWRs in human episodic memory. Our findings point to the involvement of hippocampal SWRs in establishing and triggering spontaneous recollections in the human brain. They implicate SWRs in the process of engraving new memories, and reveal their fundamental contribution in orchestrating the dialogue between memory centers (hippocampus) and high-level representations (cerebral cortex), which underlies the retrieval of these memories. Our study thus highlights the function of SWRs as powerful multitasking signals that contribute both to the encoding and to the spontaneous access and reinstatement of human memories.

Memory reactivation coupled to hippocampal ripples during free recall.

(A) Simultaneous intracranial recordings in hippocampus and cortex. (B) Patients first viewed and then freely recalled photographs of famous faces and places. (C) Rapid hippocampal neuronal bursts (SWRs) were identified (left). When patients freely recalled the images, a transient increase in SWR rate anticipated the onset of recall, dominated by items that generated a higher ripple rate (RR) during viewing (compare red and black lines). (D) Visual areas in the cortex showed SWR-coupled reactivation, recapitulating the content selectivity observed during viewing.

Abstract

Hippocampal sharp-wave ripples (SWRs) constitute one of the most synchronized activation events in the brain and play a critical role in offline memory consolidation. Yet their cognitive content and function during awake, conscious behavior remains unclear. We directly examined this question using intracranial recordings in human patients engaged in episodic free recall of previously viewed photographs. Our results reveal a content-selective increase in hippocampal ripple rate emerging 1 to 2 seconds prior to recall events. During recollection, high-order visual areas showed pronounced SWR-coupled reemergence of activation patterns associated with recalled content. Finally, the SWR rate during encoding predicted subsequent free-recall performance. These results point to a role for hippocampal SWRs in triggering spontaneous recollections and orchestrating the reinstatement of cortical representations during free episodic memory retrieval.

Hippocampal ripples (1, 2) are brief (<150 ms) high-frequency oscillatory events, in the range of 140 to 200 Hz in rodents (36) and 80 to 140 Hz in primates and humans (711), that appear in the local field potential (LFP) of the hippocampal CA1 pyramidal layer (3, 5). Conserved across a variety of species, these short-lived network oscillations constitute instances of highly synchronized neuronal activity in the brain. During a ripple, 10 to 15% of pyramidal neurons in the hippocampal-entorhinal output pathway discharge synchronously (12, 13), orchestrating a network activation that has a potent impact on several cortical and subcortical targets (10, 14). Because these ripples commonly co-occur with large-amplitude sharp waves appearing in CA1 stratum radiatum (3, 15), it is customary to refer to them as sharp-wave ripple (SWR) complexes (16). SWRs occur most frequently during non–rapid eye movement sleep and quiescent wakefulness (14, 16). In primates, SWRs can also be seen during attentive visual search, especially before the gaze is being directed toward a familiar target location (17, 18).

Electrophysiological studies of humans and rodents have demonstrated different forms of coupling between hippocampal SWRs and cortical LFP (9, 11, 1922). Extrahippocampal neuronal activations linked to previous awake experiences are reexpressed in the brief time window of the hippocampal ripple (14, 2329). The temporal relationship between SWRs and cortical reactivation during sleep suggests a coordinated bidirectional interaction whereby spontaneously generated patterns in the cortex bias the activity in the hippocampus, which then broadcasts, during the ripple, an integrated memory representation back to the cortex (24, 27). Such hippocampal-cortical interplay has been hypothesized as an orchestration mechanism that governs the reactivation of mnemonic representations across distributed cortical networks (24, 27, 30, 31).

Examination of the representational content of SWR events during ongoing awake behavior has revealed a structured, temporally compressed replay of hippocampal multicell sequences representing previous navigation-related experiences, as well as “preplay” of possible future paths (3238). Awake replay/preplay points to a potential role for SWRs in reactivating mnemonic information not only during offline consolidation, but also during ongoing awake behavior that involves recall or imagination of nonpresent scenarios (14, 3941). However, the exact cognitive content and function of SWRs during awake behavior remains unclear. This is largely because of the difficulty of assessing detailed cognitive content in animal models. Here, we addressed this challenge by using a free-recall paradigm to examine the cognitive role of SWRs in intracranial recordings of human epileptic patients.

Free recall is a cognitive process by which previously stored items are recalled spontaneously, without externally presented cueing information. It allows the dissociation between any external stimuli and the internally driven memory process. We previously showed that hippocampal and ventral-temporal neurons reactivate in a content-specific manner during free recall (42, 43). Furthermore, we were able to demonstrate a putative top-down biasing mechanism that constrains free recall to a particular category by modulating the ongoing baseline excitation of category-selective visual areas in the cortex (44).

One unique advantage of intracranial electroencephalography (iEEG) recordings conducted in patients is that the diagnostic procedure calls for multiple simultaneous recording sites in each patient. This allowed us to record LFP and SWR activity in the hippocampus simultaneously with high-frequency broadband (HFB; 60 to 160 Hz) signals reflecting local neuronal population activity (4547) in task-relevant, content-specific, cortical sites.

SWR events were recorded in patients during a resting state and during a visual free-recall task (Fig. 1A). The task consisted of two runs, each beginning with a resting-state period of 200 s. Patients were then presented with vivid, full-color photographs of famous faces and places. After viewing each picture four times in pseudorandom order and completing a short interference task, patients were instructed to freely recall the pictures, targeting each category in separate blocks. To ensure reinstatement of visual content during recall, we instructed patients to describe each recalled item with two or three prominent visual features. Verbal responses during the recall phase were recorded, and the onset and offset of each verbal recall event were carefully extracted in an offline analysis. Patients were blindfolded throughout the free-recall period to completely block external visual input.

Fig. 1 Experimental design and hippocampal SWR detection.

(A) Experimental design and stimuli (91). After viewing pictures of famous faces and places (see methods), participants were asked to freely recall and describe as many pictures as possible, targeting each category in separate blocks. (B) Coronal slice and 3D reconstruction of a hippocampal depth electrode in one representative patient. White arrow indicates CA1 recording site used for ripple detection. (C) Schematic diagram of depth iEEG electrodes used in our study. (D) Example of SWR events as they appear in the recordings. From top to bottom: raw hippocampal LFP; ripple-band filtered LFP (70 to 180 Hz); normalized ripple-band envelope used for ripple detection. (E and F) Grand average peri-ripple field potential and wavelet spectrogram centered on ripple peak (n = 8279 SWR events from 15 patients). (G) Overall distribution of inter-ripple intervals (n = 15 patients; error bars represent SEM).

On average, patients recalled 8.8 ± 2.7 (SD) items per run; when including repeated recollections, they had 12.4 ± 5.7 “recall events” per run. There was no significant difference in recall performance between the two runs (P > 0.22, Wilcoxon signed-rank test). Recall events were defined as any verbal utterance in which patients began to describe a specific picture (see methods). The average duration of verbal recall events was 8.08 ± 6.27 s. Recall performance was similar between the two categories (average number of recalled items per run: 4.43 faces, 4.37 places; P > 0.9, Wilcoxon signed-rank test).

Sharp-wave ripple detection

A multicontact depth electrode implanted in the hippocampus was used for detection of SWR events. We used pre- and postoperative computed tomography (CT) and magnetic resonance imaging (MRI) scans to identify in each patient a hippocampal recording site located in or adjacent to the CA1/CA2 subfields, where SWR events are known to occur most prominently (48). The LFP in the selected site was then filtered between 70 and 180 Hz, rectified, squared, smoothed, and transformed into z-scores. Transient events that exceeded 4 SD and survived the exclusion criteria were selected as candidate SWR events (Fig. 1D; see methods).

Figure 1B shows the location of a typical CA1 recording site in one patient; Fig. 1C is a schematic drawing of the depth electrode used in our study (locations of hippocampal recording sites in each patient are depicted in fig. S1). Figure 1D shows typical SWR events as they appear in the recordings. Our dataset included 8387 SWRs obtained from 15 patients (see Fig. 1, E and F, for grand-average peri-ripple field potential and spectral decomposition and Fig. 1G for distribution of inter-ripple interval durations; see table S1 for demographic information).

We analyzed four main conditions: (i) rest, during which patients were instructed to rest with eyes closed; (ii) viewing, during which patients inspected and memorized photographs of famous faces and places; (iii) recall, referring to times within the free-recall period in which patients verbally reported recalling a specific item from the memorized set (i.e., epochs that began 3 s before the onset until the offset of each “recall event,” a total of 269.5 s on average); and (iv) search, referring to all complementary time intervals between recall events, in which patients attempted to recall but did not report any recalled item (330.5 s on average).

SWR properties across cognitive states

We first examined whether the spectral signature of the SWRs varied across the different cognitive states of the patients, or whether it remained constant (thus reflecting oscillatory events with an all-or-none behavior). We computed a peri-ripple wavelet spectrogram in a time window of –200 to 200 ms relative to the SWR peak (see methods; see fig. S2, A to D, for traces of individual SWR events and mean spectrograms, spectra, and peri-ripple field potential across conditions). A nonparametric Friedman test comparing SWR amplitude and peak frequency showed no significant differences between the main experimental conditions [fig. S2, E and F; mean peak frequency: 88.1 ± 2.1 Hz, χ2(3) = 2.73, P > 0.43; mean peak amplitude: 9.5 ± 1.5 dB, χ2(3) = 5.00, P > 0.17; n = 15 patients]. Similar spectral characteristics have been found in sleep SWRs in humans (7, 9, 49, 50).

Having established that the basic spectral properties of the SWRs remained constant throughout the experiment, we next explored whether the SWR rate may have changed with the patients’ cognitive state. Comparing the mean SWR rate across the different experimental conditions revealed a significant effect [χ2(3) = 18.04, P = 0.0004, Friedman test, n = 15 patients; see fig. S2G], and post hoc comparisons indicated that the basal SWR rate was slightly lower during recall and memory search (i.e., inter-recall intervals) relative to the picture-viewing and rest conditions [P < 0.05, pairwise Friedman tests with false discovery rate (FDR) correction; median ripple rate (events/s): 0.45 (rest), 0.47 (viewing), 0.36 (recall), 0.34 (search)].

Content-selective modulation of SWR rate

To examine whether viewing the pictures during the encoding phase influenced the SWR rate in a more transient fashion, we computed in each patient a peristimulus time histogram (PSTH) of SWRs, showing the instantaneous SWR rate in 50-ms time bins starting from –0.5 to 2.25 s relative to picture onset (Fig. 2A). Averaging across the different pictures, we found a transient general increase in SWR rate (peaking at 675 ms poststimulus) that appeared only during the first presentation of each picture. Repeated presentations of the same pictures did not evoke this nonselective time-locked response (P < 0.01, nonparametric cluster-based permutation test, shuffling condition labels 2000 times across patients; see fig. S3, A and B, for responses across individual presentation cycles and individual patients’ data and fig. S3D for comparison between faces and places). For additional analysis examining the consistency in ripple rate across repeated presentations of the same picture, see fig. S3C.

Fig. 2 Ripples PSTH during picture viewing and free recall.

(A) SWRs raster plot and PSTH time-locked to the onset of picture presentation (n = 15 patients, each viewed 28 items × 4 presentation cycles), showing a transient increase in averaged SWR rate in response to the first but not to repeated presentations (P < 0.01, cluster-based permutation test). Black horizontal bars on the x axis represent stimulus on-periods. (B) Content selectivity of SWR rate modulation during repeated presentations, with specific images producing a higher SWR rate. Inset shows the mean rate of individual items computed over the entire stimulus period with SEM across patients. Ripple rate of high-RR images was on average 3.5 times that of low-RR images. (C) Grand-average ripples PSTH time-locked to the onset of verbal recall, showing a significant increase in SWR rate anticipating the onset of verbal report by 1 to 2 s (P < 0.01, cluster-based permutation test; see fig. S4D). (D) SWR rate during recall of high-RR and low-RR images (as defined during viewing), demonstrating recapitulation of the content selectivity observed during viewing (P < 0.05, cluster-based permutation test). Note again the anticipatory nature of the SWR rate increase. Shaded areas represent ±1 bootstrap SE computed over subjects [in (A) and (B)] or recall events [in (C) and (D)]. Gray horizontal solid/dashed lines represent mean rate (±1 SD) for the same data when SWR timing was randomly shuffled. Orange bars represent significant time bins.

SWRs elicited during viewing may have been reactivated later, in a content-specific manner, during the free-recall period, when patients recalled the same visual content but in the absence of any external stimulation. To examine this possibility, we pooled all items that were subsequently recalled in each patient (n = 252 items in total) and measured the correlation between the SWR rate evoked by each item during viewing (throughout the duration of the picture, from 50 to 1500 ms poststimulus) and the SWR rate elicited when patients freely recalled this same item (using a generic time window of 5 s centered on the onset of the verbal report of recall; repeated recollections in the same patient were averaged together). However, given the significant difference in averaged SWR rate between novel and repeated presentations described above, we analyzed the novelty-related SWRs separately from the SWRs generated during the repeated presentations.

We found a significant correlation between the SWR rates elicited by each picture during viewing and during free recall, but only for the repeated presentations—that is, when responses related to novel presentations were excluded (novel: Spearman ρ = 0.05, P > 0.43; repeated presentations: Spearman ρ = 0.18, P = 0.005; n = 252 successfully recalled items; fig. S4, A and B).

To investigate the temporal profile of this content-specific modulation of SWR rate during recall, we first sorted the pictures in each patient according to the number of SWRs they elicited during the repeated presentations (in a time window of 50 to 1500 ms poststimulus). We then divided the pictures into two groups: pictures that elicited a high SWR rate during viewing (above median), which we termed “high-RR” images; and pictures that resulted in low SWR rates (below median), which we termed “low-RR” images (Fig. 2B, inset).

Figure 2B depicts the average SWR rate when patients viewed the high-RR and low-RR images (red and dark contours, respectively). Note that the difference between these two signals is due to the selection process and is to be expected given the variable SWR responses across different images during viewing (see fig. S5 for further characterization of SWR responses across high-RR and low-RR images). The critical question is whether this content-specific difference during viewing reappeared during recall, in the absence of visual stimuli.

To answer this question, we computed for each patient a PSTH of SWRs, time-locked to the onset of verbal report of recall (using time bins of 200 ms from –5 to 5 s, smoothed with 1000-ms triangular window). Recall events with separation of less than 5 s from the previous recollection were excluded from the analysis. We first searched for a nonselective signal related to any recall event. We found a transient increase in SWR rate that preceded the onset of verbal report by 1 to 2 s (Fig. 2C). A nonparametric cluster-based permutation test, which compared the activation profile to 2000 shuffled PSTHs produced by circularly jittering SWR timing in each trial by a random amount, indicated that the anticipatory increase was highly significant (P < 0.01; cluster-defining threshold was set at ±1.96 SD from the mean rate; see also fig. S4D; significant time bins are marked in orange). Additional analysis confirmed that SWRs were not coupled to voice amplitude or instances of abrupt vocalizations (fig. S6). Movie S1 shows examples of spontaneous recall events and their relation to SWRs in three patients.

Next, we examined whether this increase during recall was content-specific. We compared SWR rates during recall of high-RR versus low-RR images (defined by the viewing responses). A nonparametric permutation test, shuffling high-RR and low-RR labels 2000 times, revealed that the SWR rate was significantly higher during recall of high-RR images (P < 0.05, see Fig. 2D; for raster plot and individual patients’ data, see fig. S4). Here, too, the content-selective increase emerged 1 to 2 s prior to the beginning of the actual verbal report.

Ripple rate during picture viewing predicts memory performance

Could SWR dynamics during picture viewing be linked to the patients’ ability to later recall these pictures? To examine this possibility, we computed a PSTH of SWRs time-locked to the onset of picture presentation, separately for the first and repeated presentations (120-ms bins smoothed by a five-point triangular window; see methods). We then computed in each patient the normalized difference in SWR rate between pictures that were later remembered or forgotten: (REM − FOR)/(REM + FOR). A cluster-based permutation test revealed that the SWR rate during picture viewing predicted the memorability of items in the subsequent free recall. Specifically, we found a higher ripple rate for remembered pictures than for forgotten pictures. This effect emerged during the poststimulus interval in the first presentation cycle (P < 0.05, one-sided cluster-based permutation test, Fig. 3, A and B; for individual patients’ data, see fig. S7). To further examine this predictive effect, we measured in each time bin the correlation between the difference in ripple rate and the patients’ memory performance during the free-recall period. We found a significant correlation, peaking during the poststimulus interval and returning back to baseline upon presentation of the next picture (P < 0.05, FDR correction; peak correlation: Spearman ρ = 0.85; Fig. 3C).

Fig. 3 Ripple rate during picture viewing predicts subsequent free-recall performance.

(A) SWRs PSTH time-locked to onset of the first presentation of each picture shows a significantly higher ripple rate for remembered versus forgotten pictures during the poststimulus interval (P = 0.02, cluster-based permutation test; n = 15 patients). (B) No significant differences were observed during repeated presentations. (C) The difference in ripple rate between remembered and forgotten items significantly predicts subsequent recall performance across patients (PFDR < 0.05; peak correlation: Spearman ρ = 0.85). Note how the correlation returns to baseline upon presentation of the next picture, attesting to the temporal specificity of this effect. (D) Left: Scatterplot showing the correlation between the poststimulus ripple rate difference (1500 to 2250 ms) and the subsequent recall performance (each dot represents an individual subject; gray line represents the least-squares fit). Right: Resampling test indicating that the correlation obtained in the actual data was highly significant (2000 iterations, P < 0.001) and did not arise from differences in the number of items in each group (remembered/forgotten). In (A) to (C), shaded area represents ±1 bootstrap SE computed over pooled trials [(A) and (B)] or patients (C); black horizontal bars on the x axis represent stimulus-on periods.

Finally, to rule out the possibility that this correlation resulted from the differences in the number of trials belonging to each group of images (remembered and forgotten), we carried out an additional permutation test, in which we shuffled the labels of the ripple rate responses 2000 times and randomly resampled the original number of remembered and forgotten trials in each patient. The results of this analysis indicated that the correlation observed in the actual data during the poststimulus interval was highly significant and did not arise from differences in the number of trials (Spearman ρ = 0.83, P < 0.001; Fig. 3D).

Ripple-triggered cortical activation

A major advantage of iEEG recordings in patients is that as a result of clinical requirements, recordings are typically obtained broadly across several cortical and medial temporal lobe (MTL) areas. We took advantage of this by comparing the activity in high-order category-selective visual electrodes (i.e., selective to face or place images) during viewing to the activity in the same recording sites during free recall, when patients recalled and verbally described the electrodes’ “preferred” and “nonpreferred” images. Image preference in each electrode was determined by sorting the different images according to the HFB amplitude they elicited during viewing (averaging the response over a time window of 100 to 500 ms and across the four presentations). We defined the top 10 items that evoked the strongest response during viewing as the “preferred” images and the bottom 10 items as the “nonpreferred” images. In most instances, the preferred and nonpreferred images also corresponded to the electrodes’ preferred and nonpreferred categories, respectively (91% of preferred images also belonged to the electrodes' preferred category, i.e., face or place). Comparing preferred versus nonpreferred items (rather than the face/place categories) enabled us to exclude “borderline” exemplars that belonged to the optimal category yet showed a weak activation, thus enhancing the sensitivity of the analysis. Furthermore, it enabled extension of the analysis to additional visual sites, whose content selectivity was significant but not necessarily related to a clear categorical division between faces and places.

Figure 4A indicates the location of cortical recording sites that showed a significant HFB (60 to 160 Hz) power increase in response to picture presentation during the viewing session [PFDR < 0.05, Wilcoxon signed-rank test comparing stimulus response (100 to 500 ms) versus prestimulus baseline (–400 to –100 ms)]. Visual electrodes that showed a preferential HFB response to pictures of faces or places were regarded as category-selective (PFDR < 0.05, Wilcoxon rank-sum test, faces versus places; see methods). They were typically localized in high-order visual areas along the ventral visual stream, lateral and medial to the fusiform gyrus.

Fig. 4 Visually responsive electrodes and examples of peri-ripple visual reactivation during recall.

(A) Multipatient electrode coverage showing the location of visually responsive electrodes in relation to early visual areas (blue), intermediate visual areas (yellow), and the fusiform gyrus (pink). Face- and place-selective bipolar electrode pairs (bipoles) are colored red and green, respectively. Each dot represents a single electrode contact taking part in a bipole. Note the clear tendency of category-selective electrodes to be localized in high-order visual areas along the ventral stream. (B to D) Category-selective peri-ripple HFB response during recall in three representative category-selective sites. Left: Anatomical location of each recording site. Center: HFB response time-locked to the onset of picture presentation (comparing face versus place images; orange bar represents significant time clusters at P < 0.05). Right: Peri-ripple HFB response during the verbal report of recall, when patients freely recalled the same face or place images (orange bar represents significant time points at P < 0.05, Wilcoxon rank-sum test). Note the category-selective modulation of HFB power around the simultaneously recorded hippocampal ripple. As can be seen in these examples, the peri-ripple cortical response involves either amplitude increase or decrease, depending on the recalled content and whether it matches the preferred representational content of the recorded site. Black bars on the x axis represent stimulus-on periods; shaded areas represent SEM.

To examine the potential role of SWRs in coordinating reactivation of cortical representations during recall, we time-locked the activity in category-selective visual sites to the onset of hippocampal SWRs. Specifically, we examined whether those cortical sites were reactivated during recall-related hippocampal SWRs and whether the reactivation was content-specific (i.e., matched content preference during viewing). Shown in Fig. 4, B to D, are HFB responses in three representative category-selective recording sites during picture viewing and free recall. During recollection, these electrodes showed a small, transient modulation of HFB amplitude time-locked to the onset of hippocampal SWR events. Critically, this coupled cortical activity was content-specific; that is, it occurred only when the patients recalled images from the electrodes’ preferred category (preference that was revealed during viewing). In some cases, recalling the nonpreferred category led to a decrease in HFB amplitude. Thus, cortical activity coupled to hippocampal SWRs appeared most prominently when contrasting the preferred versus the nonpreferred images in each recording site.

To examine whether this peri-ripple amplitude modulation was a general phenomenon across the entire group of category-selective visual electrodes, we computed a multitaper spectrogram for each recording site during a time window of –750 to 750 ms relative to SWR onset (see methods). When patients recalled the electrodes’ preferred images (i.e., top 10 images that elicited the strongest response during viewing), there was a small but highly consistent HFB activation centered around the onset of hippocampal SWRs (Fig. 5, A to C, PFDR < 0.001, n = 57, Wilcoxon signed-rank test; peak normalized amplitude, 0.24 dB; SE, ±0.04). This peri-ripple cortical response involved a broadband power increase in frequencies between 50 and 180 Hz (High-Gamma), a signal known to reflect a local increase in population firing rate (47, 51). This peri-ripple visual activation was significantly higher when patients recalled the electrodes’ preferred images (top 10 images) as compared to the nonpreferred ones (bottom 10 images that least activated the electrodes during viewing) (P < 0.01, cluster-based nonparametric permutation test, shuffling preferred/nonpreferred labels 2000 times over electrodes; n = 57 recording sites in 13 patients, after excluding recording sites with fewer than five peri-ripple responses in each condition). Lower frequencies (1 to 30 Hz) did not exhibit content-selective power changes (no significant differences; preferred versus nonpreferred images, cluster-based permutation test; fig. S8A).

Fig. 5 Peri-ripple reactivation across the visual hierarchy during free recall.

(A) HFB response to preferred (top 10) and nonpreferred (bottom 10) images in face/place selective recording sites during viewing. (B) HFB activity in the same category-selective electrodes during recall, time-locked to the onset of simultaneously recorded hippocampal ripples. Note the selective transient increase in HFB power during recall of preferred versus nonpreferred images (P < 0.05, cluster-based permutation test; n = 57 bipoles from 13 patients). (C) Multitaper spectrograms showing that peri-ripple cortical responses were concentrated in a broad high-frequency range (50 to 180 Hz). (D) Reactivation effect size (Hedges’ g) in visually responsive electrodes, comparing peri-ripple responses during recall of “preferred” versus “nonpreferred” images. (E and F) Percentage of significant electrodes (P < 0.05, Wilcoxon rank-sum test, uncorrected) and mean effect size in each region of interest. Peri-ripple response selectivity was strongest in the fusiform gyrus and entorhinal cortex. Note the clear tendency for increased reactivation effect at more anterior-medial sites. Error bars and shaded areas represent SEM. In (A) and (B), horizontal orange bars represent significant time clusters.

Was the SWR-triggered effect specific to overtly reported recall events? We performed the same analysis on SWRs that occurred while patients attempted to recall the electrode’s preferred and nonpreferred categories but did not overtly report any recalled item (i.e., the “memory search” period). There was no content-selective peri-ripple activation during these inter-recall periods (no significant differences; fig. S8, B to D); this finding suggested that the effect was specific to conscious, reportable recall events (further comparisons among recall, memory search, and resting-state SWRs are depicted in fig. S8F). Finally, using a bootstrap sampling procedure with 2000 resamples, we estimated the latency of the maximal difference between the preferred and nonpreferred spectrograms during a [–300, 300] ms time window centered on SWR onset. The analysis showed a slight trend of an advance cortical activation [mean peak latency: –18 ms, 95% CI (–65, 29); frequency: 102.1 Hz, 95% CI (85, 118)]; however, this effect was not statistically significant.

We next examined how content-selective peri-ripple activation was distributed across the entire set of visually responsive electrodes. Figure 5D depicts the distribution of all recording sites in our dataset presented on an average cortical template. Recording sites that showed a significant visual response during the picture-viewing condition were color-coded according to their peri-ripple reactivation effect during recall. To obtain this map, we first identified the 10 images that produced the strongest and weakest responses during viewing individually in each recording site. Then, we computed a peri-ripple spectrogram during the verbal recall periods (similar to the analysis in Fig. 5, B and C). We separated the activations that occurred when patients recalled the electrode’s preferred images from those that occurred when the patients recalled the nonpreferred ones. We then averaged the HFB power over a [–300, 300] ms time window centered on SWR onset and quantified the difference between preferred and nonpreferred images using the bias-corrected Hedges’ g effect size measure (52), individually in each site. Recording sites with fewer than five peri-ripple responses in either the preferred or nonpreferred group were excluded. Content-selective reactivation of visual information during SWR events occurred most prominently in the fusiform gyrus as well as in downstream cortical regions including the entorhinal and perirhinal cortices. We used the anatomical atlases (5355) included in FreeSurfer to subdivide the cortical surface into six partly overlapping regions along the visual hierarchy (56). Electrodes falling within these regions were grouped together. The peri-ripple reactivation was strongest in the fusiform gyrus and entorhinal cortex (i.e., the higher levels of the ventral visual hierarchy) (Fig. 5, E and F).

Memory reinstatement during SWRs

To further investigate the reinstatement of visual information during recall in relation to SWRs, we pooled together visually responsive cortical sites that showed significant content selectivity during picture viewing (see methods; see fig. S9 for electrodes’ location) and constructed a multisite HFB activation pattern per each item during picture viewing and recall. We used principal components analysis (PCA) to reduce the dimensionality of the activation patterns during viewing, retaining the first 11 principal components (PCs) that accounted for 83.8% of the variation across viewed items (see methods; see Fig. 6, A and B, for PC visualization). We then applied the same linear transformation to the patterns that emerged during recall, bringing all patterns to the same 11-dimensional linear space. Next, we quantified the similarity between viewing and recall patterns. We computed the Pearson correlation using a 50-ms sliding window to examine how the similarity between patterns changed in relation to SWR timing during recall, and in relation to the onset/offset of the picture during viewing (Fig. 6, C and D). This analysis revealed a significant enhancement in pattern correlation during the SWR event (P < 0.05, nonparametric cluster-based permutation test; see methods). Intriguingly, there was a trend toward a second peak shortly after the disappearance of the picture; however, this peak did not survive the cluster-based permutation test.

Fig. 6 Similarity between recall decoding performance and cortical activation patterns during viewing and free recall.

(A) PCA applied to multivariate activation patterns during picture viewing. Note that the first principal component (accounting for 40% of the variance) captured the categorical difference between items. (B) Visualizing the patterns according to PC1, PC2, and PC3 showed a clear categorical clustering of faces and places, with some additional differentiation at the item level. (C) After dimensionality reduction, Pearson correlation was used to quantify the similarity between viewing and recall patterns, with a 50-ms sliding window (91). (D) Viewing-recall pattern similarity relative to the SWR event and to the onset/offset of the pictures (black horizontal lines). Significant correlation was found only during the SWR event (P < 0.05, cluster-based permutation test; significant clusters are contoured in black). (E and F) Performance of k-NN classifier trained on viewing patterns and tested on recall patterns (cross-classification). Shaded gray area shows the decoding performance for shuffled data (mean ± SD). Dashed orange lines represent the cluster-defining threshold (P = 0.05); significant time clusters are marked above (P < 0.01, cluster-based permutation test). For visualization, decoding performance was smoothed using a boxcar filter three time bins wide.

We then asked whether we could decode the identity of recalled items on the basis of the SWR-triggered cortical HFB patterns. To perform this analysis, we trained a k-nearest neighbors (k-NN) classifier on single-trial HFB patterns during viewing and tested its classification performance on the free-recall patterns (i.e., cross-classification; see methods). Here again, the patterns’ dimensionality was reduced using an out-of-sample extension of the same PCA transformation described above (Fig. 6, A and B).

Testing the classifier performance on the viewing data showed 100% accuracy in decoding the image category (using k = 9 NN), and 47.3% accuracy (chance level is 3.5%) in decoding exemplar identity (using k = 1 NN). For the cross-classification analysis, we used a 50-ms sliding window to examine the temporal profile of visual reinstatement during recall relative to the SWR onset (Fig. 6, E and F). We obtained significant decoding performance during recall for both category (82.1% accuracy, 23/28 items) and exemplar identity (21.4% accuracy, 6/28 items) (P < 0.01; a nonparametric cluster-based permutation test, shuffling item labels 2000 times; see methods). Decoding performance peaked together with the SWR event, suggesting a temporally precise coupling between hippocampal SWRs and cortical activity during reinstatement of visual information.

Discussion

We used a rare clinical opportunity to measure hippocampal SWRs and the associated SWR-triggered cortical activity in human patients as they memorized and freely recalled vivid photographs of famous faces and places. Our results highlight three major new aspects of SWRs’ function and their relation to human episodic memory. First, a transient increase in hippocampal SWR rate preceded the onset of verbally reported recollections by 1 to 2 s. This increase was content-selective and reexpressed the same picture preferences observed during the encoding phase. Second, the SWR rate during picture viewing predicted subsequent memory performance of individual patients. Finally, during the verbal report of recall, high-order cortical visual sites showed a SWR-triggered increase in HFB activity. Again, this broadband activation was content-specific and occurred only when the patients recalled the pictures that preferentially activated the sites during viewing.

The anticipatory increase in SWR rate during recall (Fig. 2C) strongly suggests that SWRs play an important role in the initiation of self-generated recall events. Work in rodents exploring the link between awake SWRs and putative memory retrieval behaviors has demonstrated that sequences of hippocampal place cell assemblies, representing spatial and contextual information related to past experiences, are briefly replayed in the time window of the SWR (32, 34, 35, 41, 57). However, it was not possible to determine in these studies the actual moment of cognitive recall, and hence its temporal relationship to SWR events. Conducting the experiment in awake human patients enabled us to extend these previous studies by obtaining an estimation of when each recalled item surfaced into the patients’ conscious awareness. This allowed us to establish the anticipatory nature of the SWR event.

It may be argued that this anticipatory increase could result from inaccuracies in the timing of verbal reports. However, because the SWR rate increase was transient and clearly declined at the time of the verbal report proper, such an onset “blurring” effect is unlikely. These results are intriguingly similar to the anticipatory increases in hippocampal and medial temporal neurons’ firing rate observed in single-unit recordings in patients during a similar free-recall paradigm (42). These anticipatory hippocampal signals are compatible with a two-stage recollection process mediated by the hippocampus: a fast subconscious stage, involving reactivation of hippocampal-neocortical memory traces, and a slower conscious one, involving cortical processes that operate on the retrieved content and reinstate the mentally experienced episode (58). However, we cannot at this point rule out the possibility that patients thought about the recalled items prior to their verbal responses, hence contributing to the anticipatory activation.

The increase in SWR rate prior to recall onset showed visual content selectivity: Specific images that generated a higher SWR rate during the picture-viewing stage also elicited a higher SWR rate during recollection. In other words, the SWR rate during recall reexpressed the content specificity found during viewing. Thus, the phenomenon of memory reactivation is evident not only in the spike content of the SWR (e.g., content-specific sequences of hippocampal place cells) (3235, 57, 59) but also in the rate of SWRs elicited during recall, which is linked to the rate of SWRs elicited during the original experience. However, it should be noted that experiences that are encoded for the first time are likely to engage a different set of memory processes (e.g., novelty detection, engram formation, etc.) that do not repeat during recall. In line with this, the link between SWR rates during the original experience and subsequent recall was found only for the repeated item presentations (fig. S4).

Our results show that SWRs play an important role in the encoding process as well. We found that the ability of patients to successfully recall a visual item was significantly linked to SWR activity during picture viewing (memory-encoding stage). This effect was observed only during the first presentation of each image and was maximal during the postpresentation period, thereby corroborating previous studies pointing to the importance of the poststimulus periods in memory encoding (60). Specifically, we found that the strength of the differential signal during encoding (i.e., the difference in SWR rate after presentation of recalled and forgotten items) predicted the success of patients in subsequently recalling these items. A plausible interpretation of this effect is that such differential signal during viewing may capture the process of memory trace formation, so that the larger the differential activity, the stronger the engagement of the hippocampus in the encoding process, which enhances the ability of the patient to freely recall these memories later on. Regardless of the precise mechanism, this effect implicates the involvement of SWRs in memory formation.

Our observation that SWRs tend to emerge rather frequently during a visual memory task that is clearly nonspatial in nature demonstrates that SWRs are not exclusive to navigational aspects, but rather play a more general role in episodic memory (14). This is compatible with the occurrence of SWRs during a visual search task in primates (18).

Finally, an important aspect of SWR function uncovered by the current study is the content-selective coupling between SWRs and cortical activation. This content selectivity, reflected in HFB activity in high-order cortical sites, was precisely time-locked to the SWR event itself. Thus, SWR-triggered activity in high-order visual sites was significantly higher when patients recalled items that preferentially activated these sites during picture viewing. Reactivation of visual content occurred most prominently during the time window of the SWR. The SWR-triggered cortical activity was specific to the actual recall events and was not found during the search times between verbal recalls. This result further attests to the specific role of this activity in reportable, conscious recollection.

Previous work in rodents (27, 61) has demonstrated a bidirectional interaction between the hippocampus and the cortex during memory consolidation and retrieval. Such studies suggested a role for the cortex in facilitating reactivation of the relevant hippocampal representation during a SWR. Consistent with these suggestions, previous work in humans (43) has demonstrated a slow, anticipatory activation of category-specific cortical information that precedes the actual moment of recall by several seconds (44, 62). Moreover, during internal memory search, when subjects attempt to recall a particular category but fail to come up with a specific exemplar, activity in category-specific cortical sites remains slightly elevated, possibly reflecting a top-down control signal that imposes categorical boundaries on downstream memory representations in the hippocampus (44). Our finding of a slight trend of an advanced cortical activation prior to SWR onset is compatible with the suggested top-down cortical influence and the bidirectional nature of the hippocampal-cortical interplay in general, although further research will be required to fully characterize and confirm this interaction.

Our results are consistent with recent work by Vaz and colleagues (63) showing coupled ripple-band activity in MTL and temporal association cortex during successful retrieval. Both spectral and temporal profiles of these coupled oscillatory events are compatible with the SWR-triggered HFB activation we observed.

Consistent with rodent studies reporting higher SWR rates during exploration of novel environments than of familiar ones (14), we observed a significantly higher increase in average SWR rate in response to the first presentation of each picture than for subsequent presentations of the same pictures. Unlike SWR activity evoked by repeated presentations, SWR rates during novel presentations were not reinstated during the subsequent recall. This pattern of results suggests the involvement of two different subtypes of SWRs elicited during viewing: one that reflects the general processing of novel information and another that reflects a mnemonic process that is more content-specific in nature (i.e., recognition memory, pattern retrieval, etc.). However, it is important to note, more generally, that processes of memory retrieval may be closely linked to processes of memory consolidation (and reconsolidation), so the observed changes in ripple rate may underlie both processes (14).

Together, our results demonstrate an important link between SWRs and verbally reported human episodic memory. More specifically, they reveal SWR-related reinstatement of visual representations during free recall. These results point to a process by which SWRs set up an integrated, content-specific dialogue between the hippocampus and the cortex that initiates and enables the process of recall.

Methods

Participants

Intracranial recordings were obtained from 15 patients with pharmacologically resistant epilepsy (10 females) at the North Shore University Hospital, New York. The age of the patients ranged from 22 to 57 (mean = 36.6, SD = 10.7). All patients were implanted with subdural intracranial electrodes for diagnostic purposes as part of their evaluation for neurosurgical epilepsy treatment. All participants performed the task in their native language (12 English speakers, three Spanish speakers). No clinical seizures occurred during the experimental duration. The study was conducted according to the latest version of the Declaration of Helsinki, and all participants provided a fully informed consent according to NIH guidelines, as monitored by the institutional review board at the Feinstein Institute for Medical Research.

Experimental task

The experiment was divided into two runs. Each run began with a closed-eyes resting-state period of 200 s (the first two patients performed the resting state on a different day). Immediately afterward, participants were presented with 14 different pictures of famous faces and places (seven in each category; see Fig. 1 for example stimuli). Picture duration was 1500 ms with 750-ms interstimulus intervals. Each item repeated four times in a pseudorandom order, such that each presentation cycle contained all pictures but the order of pictures was randomized within the cycle. The same picture was never presented twice consecutively. Participants were instructed to look carefully at the pictures and try to remember them in detail, emphasizing unique colors, face expressions, perspective, lighting, etc. Stimuli were presented on a standard LCD screen using Presentation software (picture size: 16.5° × 12.7° at ~60 cm viewing distance). After viewing the pictures, participants put on a blindfold and began a short interference task of counting back from 150 in steps of 5 for approximately 1 min. Upon completion, recall instructions were presented. The patients were asked to freely recall as many pictures as possible while focusing on one category at a time, starting with faces in the first run and with places in the second run.

We instructed the patients to describe each picture they recalled, as soon as it came to mind, with two or three prominent visual features. This was done to ensure that the patients also retrieved episodic visual information specific to the studied items, and not just general semantic details. The duration of the free-recall phase was 2.5 min per each category (5 min in total × two runs). In case the patients indicated that they were “through,” they received a standard prompt from the experimenter (e.g., “Can you remember any more pictures?”). Each run included a new set of pictures, and the order of recalled categories was counterbalanced between the runs.

Identification of verbal recall events

Verbal responses during the free-recall phase were continuously recorded using a microphone attached to the patient’s gown. The onset and offset of each recall event were extracted in an offline analysis, identifying the first/last sound wave relevant to each utterance (44), using Audacity recording and editing software (version 2.0.6). SWR events occurring during the verbally reported recall events, or in the 3 s that immediately preceded the events, were associated with the item that the patient described. SWR events that occurred in between recall events were regarded as “memory search” ripples and were associated only with the category that patient was instructed to recall in the beginning of the free-recall block.

Intracranial recordings

Intracranial recording sites were subdural grids, strips, or depth electrodes (Ad-Tech, Racine, WI; Integra, Plainsboro, NJ; PMT Corporation, Chanhassen, MN). Recording sites in the subdural grids and strips were 1- or 3-mm platinum disks with 4- or 10-mm intercontact spacing. Recording sites in depth electrodes implanted in the hippocampus were 2-mm platinum cylinders with 4.4-mm intercontact spacing and a diameter of 0.8 mm (see Fig. 1C). During the recordings, the intracranial EEG signal was referenced to a vertex screw/subdermal electrode and was filtered electronically between 0.1 and 200 Hz. The signal was then digitized at 500 Hz/512 Hz and stored for offline analysis using XLTEK EMU128FS/NeuroLink IP 256 systems (Natus Medical Inc., San Carlos, CA). Stimulus-triggered electrical pulses were recorded along with the iEEG data for precise synchronization with stimulus onset. All recordings were conducted at the patients’ quiet bedside.

Electrode localization

Prior to electrode implantation, we obtained for each patient a T1-weighted 1-mm isometric structural MRI scan using a 3-T scanner. After implantation, a CT scan and a T1-weighted structural MRI scan at 1.5 T were acquired. The post-implantation CT and MRI scans were skull-stripped and co-registered to the preoperative anatomical MRI scan using FSL’s BET and FLIRT algorithms (6466). Concatenating these two co-registrations allowed visualization of the CT scan on top of the preoperative MRI scan while minimizing localization error due to potential brain shift caused by surgery and implantation. Individual recoding sites were then identified visually on the co-registered CT and were marked in each subject’s preoperative MRI native space using BioImage Suite (67).

Next, preoperative structural MRI scans were processed using FreeSurfer 6.0 (68) to segment and reconstruct the cortical surface and hippocampal subfields in each patient. Following our previously published procedure (44, 69), the three-dimensional mesh of the cortical surface in each patient was resampled and standardized using SUMA (70), allowing us to establish node-to-node correspondence across different surfaces. This enabled us to visualize electrodes from different patients on a single cortical template (“fsaverage”) while adhering to the electrodes’ location in relation to individual gyri and sulci. Finally, each cortical surface was registered onto different anatomical atlases (54, 55) available in FreeSurfer, including a probabilistic atlas of visual retinotopy (53).

Preprocessing and data analysis

All data analysis was performed in MATLAB 2014a/2018b (MathWorks Inc., Natick, MA) using EEGLAB (71), Chronux (72), DRtoolbox (https://lvdmaaten.github.io/drtoolbox/), MES toolbox (52), and custom-developed analysis routines. Raw iEEG data were inspected visually and statistically to detect noisy/corrupted channels and exclude them from further analysis. The preprocessing began by converting the iEEG signals to bipolar derivations by pairing adjacent electrode contacts. Recording sites in the hippocampus were paired with a nearby white-matter electrode that was identified anatomically using FreeSurfer’s segmentation (68). We then resampled each bipolar derivation at 500 Hz and removed the 60-Hz power line interference (including its harmonics) using zero-lag linear-phase Hamming-windowed FIR band-stop filters (3 Hz wide).

High-frequency broadband signal and spectral analysis

HFB signal was defined in the present study as the mean normalized power of frequencies between 60 and 160 Hz (High-Gamma). This range of frequencies was used as the key electrophysiological marker of local neural population activity (4547, 73). For analyses in which spectrograms were computed, HFB power was calculated as the average of frequency rows between 60 and 160 Hz. In all other cases, HFB power was computed by filtering the signal in 20-Hz bands between 60 and 160 Hz (using zero-lag linear-phase Hamming-windowed FIR filters) and calculating the normalized, 1/f corrected, analytic amplitude using a Hilbert transform (44). The latter method was mainly used for detecting visually responsive/category-selective recording sites and estimating their response latency [see (44) for details].

HFB data were inspected for transient electrical artifacts, defined as peaks above 5σ that appear in the HFB time series of the common average signal (i.e., the average LFP across all iEEG channels). Time windows of 200 ms around these peaks were logged for exclusion in subsequent analyses.

Spectral decomposition of SWR events in hippocampal recording sites was done using a Morlet-wavelet time-frequency method, implemented in EEGLAB. We used a window of 1 cycle at the lowest frequency (4 Hz) and up to 20 cycles at the highest plotted frequency (220 Hz), with a step size of 4 ms. Ripple-triggered spectrograms were normalized by the geometric mean power in each frequency, computed over the entire epoch length (–750 to 750 ms) and across all epochs belonging to the same condition (i.e., rest, picture viewing, recalling faces, recalling places) in a given run.

Spectral decomposition of HFB activation in cortical recording sites was done using the multitaper method (74) implemented in Chronux (http://chronux.org/) (75). For analysis of frequencies above 30 Hz, we used a combination of five tapers and a 200-ms-wide time window (advanced in 6-ms steps), resulting in frequency resolution of 20 Hz. For analysis of frequencies below 30 Hz, we used a combination of three tapers and a 500-ms-wide time window (advanced in 10-ms steps), resulting in frequency resolution of 5 Hz. Here again, the ripple-triggered spectrograms were normalized by the geometric mean power in each frequency, computed over the entire epoch length (–750 to 750 ms) and across all epochs belonging to the same condition in a given run. Stimulus-triggered spectrograms in the picture-viewing stage were normalized relative to a baseline period of –400 to –100 ms prestimulus.

Visually responsive electrodes

We identified visually responsive sites by comparing, in each bipolar electrode pair, the poststimulus HFB response (averaged over a time window of 100 to 500 ms) to the prestimulus baseline (–400 to –100 ms) using a two-tailed Wilcoxon signed-rank test. P values from all recording sites (across all patients) were pooled together to control the FDR (76). Bipoles that showed a significant HFB response (PFDR < 0.05) were regarded as visually responsive. Visual bipoles that were fully contained within Brodmann areas 17/18 (V1/V2), and exhibited response latency shorter than 180 ms were labeled “early visual”. To define face-selective and place-selective bipoles, we averaged the visual HFB responses over a time window of 100 to 500 ms poststimulus and compared faces versus places using a Wilcoxon rank sum test. Significant bipoles (PFDR < 0.05) located beyond early visual areas (V1/V2) were labeled either “face-selective” or “place-selective,” correspondingly. The remaining visually responsive bipoles were grouped together according to their anatomical/retinotopic location (5355). When assigning bipoles to a specific region, we only required that one of the two contacts be located within that region, thus allowing for the same bipole to be attributed to two different regions (in cases where the bipole was located on the border between regions).

Offline ripple detection

Ripple detection was performed using a macro electrode contact located in or adjacent to the CA1/CA2 subfields, as identified anatomically in each patient using FreeSurfer’s hippocampal subfields parcellation algorithm (77) (the exact anatomical location in each patient is depicted in fig. S1). For technical reasons, ripple detection in two of the patients was performed using a contact located in the subiculum [where SWR events can also be clearly identified (12)]. Prior to ripple detection, a reference signal from a nearby white-matter contact was subtracted to eliminate common noise. LFPs were then filtered between 70 and 180 Hz (zero-lag linear-phase Hamming windowed FIR filter with a transition bandwidth of 5 Hz) and instantaneous analytic amplitude was computed using a Hilbert transform. Following the procedure of (78), extreme values were clipped to 4 SD to minimize ripple rate–induced biasing. The clipped signal was then squared and smoothed (Kaiser-window FIR low-pass filter with 40 Hz cutoff), and the mean and SD were computed across the entire experimental duration to define the threshold for event detection. Events from the original (squared but unclipped) signal that exceeded 4 SD above baseline were selected as candidate SWR events. Event duration was expanded until ripple power fell below 2 SD. Events shorter than 20 ms or longer than 200 ms were excluded. Adjacent events with less than 30 ms separation (peak-to-peak) were merged. Finally, SWR peak was aligned to the trough (of the nonrectified signal) closest to the peak power.

A control detection was performed on the common average signal computed across all iEEG channels. Hippocampal SWR events that coincided with common average ripple-band peaks were removed, thus avoiding erroneous detection of transient electrical and muscular artifacts that tend to appear simultaneously on multiple channels (79, 80).

Lastly, to avoid inclusion of possible pathological events, we removed any SWR events that occurred within 50 ms from inter-ictal epileptic discharges (IEDs) (81). The latter were detected by filtering the raw hippocampal LFP between 25 and 60 Hz (zero-lag linear-phase Hamming windowed FIR filter), and similar to the above procedure, rectifying, squaring, smoothing, normalizing, and detecting events that exceeded 4 SD.

The frequency window used for ripple detection in the present study was based on previous research in humans (7, 8, 11, 82), pointing to a typical ripple-band frequency range of 80 to 140 Hz, that might occasionally reach up to 170 Hz in individual events. Thus, to minimize the possibility of filtering out genuine ripples, we used a frequency range of 70 to 180 Hz (taking into account the filter roll-off). Notably, selecting a narrower filter (e.g., 70 to 130 Hz) did not introduce any substantial changes to the main results.

SWR peristimulus time histogram

We used the following parameters to construct PSTHs of SWR events across the different experimental conditions. For picture viewing responses (Fig. 2A), we used 50-ms time bins starting from –0.5 to 2.25 s relative to picture onset, smoothed by a 5-point triangular window. To compare ripple rate between remembered and forgotten items (Fig. 3), we used a bin width of 120 ms, based on Scott’s optimization method (83), to accommodate the lower number of trials. To construct PSTH during recall events, we used a bin size of 200 ms, smoothed by a 5-point triangular window.

Multivariate pattern analysis (MVPA)

Multivariate HFB activation patterns were constructed by pooling visually responsive recording sites from all subjects. For the analysis, we first defined six regions of interest along the ventral visual hierarchy, using the Desikan-Killiany atlas (54), including the lateral occipital cortex (LO), inferior temporal gyrus (ITG), lingual gyrus, parahippocampal gyrus (PHG), fusiform gyrus, and entorhinal cortex. Visually responsive electrodes that fell within these anatomical regions and showed a substantial content selectivity in their responses during picture viewing [i.e., a difference of at least 3 SD between preferred (top 10) and nonpreferred (bottom 10) images] were included in the analysis (n = 78 bipolar electrode pairs; electrodes’ location is depicted in fig. S9).

To construct HFB activation patterns associated with the viewed images, we first computed in each recording site the instantaneous HFB power using multitaper spectrograms (as described above). In the picture-viewing condition, spectrograms were computed in a time window of –250 to 2250 ms relative to picture onset. In the free-recall condition, spectrograms were time-locked to hippocampal SWR events that occurred during the verbal report of recall (from –500 to 500 ms relative to ripple onset). Each SWR event was uniquely associated with the picture the subject was describing at the time of the event.

Next, we applied z-score transformation to the HFB power in each recording site. Z-scores were computed across all items within the same run, individually for each time point. Then, the instantaneous power was binned in 50-ms time bins (using 80% overlap). For the viewing data, power values were averaged across the four presentations of each picture, resulting in a matrix of 28 items × 78 bipoles × 236 time bins. For the recall data, power values were averaged across all SWR events associated with the same item, resulting in a matrix of 28 items × 78 bipoles × 106 time bins. For items that were not recalled by a certain patient, or did not elicit ripples, the corresponding (missing) entry in the matrix was replaced by zero (i.e., the mean).

We next averaged the viewing data over the entire stimulus duration (from 100 to 1,500 ms) to construct a “template” feature matrix of visual responses (28 items × 78 bipoles), and applied PCA to reduce the dimensionality of the features (84). To determine the number of PCs to retain, we estimated the true dimensionality of the data (i.e., intrinsic dimension) using a maximum likelihood estimation technique (85). This led us to retain the first 11 PCs, which explained 83.8% of the variation in the data (see Fig. 6, A and B).

To compute the similarity between cortical patterns that emerged during viewing and peri-ripple cortical patterns that emerged during recall, we brought all instantaneous patterns (binned in 50-ms time bins) to the same linear space by reapplying the same linear transformation that was obtained from the PCA of the averaged feature matrix described above (i.e., an out-of-sample extension of the PCA). The same linear mapping was applied to both viewing and free-recall patterns.

Finally, we used Pearson correlation to quantify the similarity between viewing and recall patterns in each 50-ms time bin. This was done to examine how the correlation changed relative to the SWR onset (during recall) and relative to the onset/offset of the picture (during viewing) (Fig. 6D). To assess statistical significance, we performed a nonparametric cluster-based permutation test, shuffling item labels 2000 times, recomputing the correlation values, and measuring the maximal cluster size after applying a threshold of P < 0.01 on the correlation values.

Cross-classification analysis

To test whether we could decode the identity of recalled items from the ripple-triggered cortical HFB patterns, we trained a k-nearest neighbors (k-NN) classifier on single-trial viewing patterns (28 items, each presented four times, n = 112 trials in total) and tested its classification performance on the peri-ripple patterns that emerged during the verbally reported recall events (i.e., cross-classification analysis; Fig. 6, E and F). For this analysis, patterns elicited during viewing were averaged over a time window of 100 to 500 ms poststimulus [where visual responses are strongest and most informative about stimulus identity (86, 87)]. Here again, we reduced the dimensionality of the data using PCA, and applied the same transformation also to the peri-ripple patterns during recall, as described above for the MVPA (i.e., an out-of-sample extension of the same PCA that was applied on the viewing patterns). We used k = 9 nearest neighbors to decode image category and k = 1 nearest neighbors to decode exemplar identity. To measure classification performance in the viewing condition, we used a leave-one-out cross-validation technique. For the cross-classification analysis, decoding the recalled content, we computed classification accuracy individually in each 50-ms time bin, from –500 to 500 ms relative to the onset of the hippocampal SWR event. Statistical significance was assessed using a nonparametric cluster-based permutation test, shuffling item labels 2000 times and recomputing the cross-classification performance while measuring the size of the maximal cluster in each iteration (using a cluster-defining threshold of P = 0.05). FWE-corrected P values were computed as the proportion of random clusters larger than or equal to the clusters observed in the actual data.

Statistical analyses

For statistical testing, parametric methods were used for normal data. Because HFB amplitude, like other measures of population firing rate, tends to follow a log-normal distribution, amplitude values were log-transformed into decibel (10 × log10) prior to any statistical testing. For non-normal data or small sample sizes, we used Wilcoxon signed-rank/rank sum tests. All statistical tests were two-sided unless stated otherwise. A Greenhouse-Geisser correction for sphericity was applied to repeated-measures analyses of variance when necessary. Multiple-comparisons correction was performed either through the Benjamini-Hochberg method (76) for FDR adjustment or by using nonparametric cluster-based permutation tests developed by others (88, 89), in which the family-wise error rate (FWE) is inherently controlled (90). No statistical methods were used to predetermine sample sizes; however, sample sizes were similar to those generally used in the field. Data collection and analysis were not performed blind to the conditions of the experiments.

Supplementary Materials

References and Notes

  1. These six regions are the early and intermediate visual areas (including V1/V2/V3/V4 and other retinotopic areas along the dorsal stream), lateral occipital cortex, inferior temporal gyrus, lingual/parahippocampal gyrus, fusiform gyrus, and entorhinal cortex.
  2. Images shown to patients are protected by copyright. Images presented in the figure are similar substitutes. Photo of Bill Clinton courtesy of Gage Skidmore; photo of the Golden Gate Bridge courtesy of Nicolas Raymond; photo of the Leaning Tower of Pisa courtesy of Josu; photo of Barack Obama courtesy of the U.S. government. All pictures are published under a Creative Commons license.
Acknowledgments: We are grateful to the patients for their kind cooperation. Y.N. thanks his wife and his daughter for their continuous and loving support. We thank R. Amit, O. Sharon, R. Broday-Dvir, and S. Grossman for intellectual support, encouragement, and helpful feedback. Funding: Supported by U.S.-Israel Binational Foundation grant 2017015 (R.M. and A.D.M.) and a CIFAR Tanenbaum Fellowship (R.M.). Author contributions: R.M. and Y.N. conceived the study and designed the experiment. Y.N. analyzed the data. R.M. supervised the analysis. E.M.Y. and S.K. ran the experiments. A.D.M. performed the surgeries and supervised the experiments and all aspects of data collection. M.H. and E.M.Y. contributed to electrode localization. Y.N. and R.M. wrote the paper. A.D.M., S.K., and E.M.Y. further contributed to the writing by reviewing and editing the manuscript. Competing interests: The authors declare no competing financial interests. Data and materials availability: The data and code that support the conclusions of this study are available upon reasonable request from Y.N. (itzik.norman@gmail.com) and are also accessible online at Zenodo (https://doi.org/10.5281/zenodo.3259369).
View Abstract

Stay Connected to Science

Navigate This Article