Cortical information flow during flexible sensorimotor decisions

See allHide authors and affiliations

Science  19 Jun 2015:
Vol. 348, Issue 6241, pp. 1352-1355
DOI: 10.1126/science.aab0551

Signal flow during sensorimotor choices

Little is known about the flow of task signals across the brain. Siegel et al. simultaneously recorded from multiple units in the sensory, parietal, prefrontal, and motor cortex while monkeys were cued to perform one among two possible simple tasks. The proportion of neurons coding for stimuli, cues, tasks, and choices, and their response latency, varied across regions. Parietal and prefrontal brain regions encoded task information and choices with the same latency. Interestingly, all brain areas encoded all types of information. However, they differed functionally according to the proportions of neurons and their response latency.

Science, this issue p. 1352


During flexible behavior, multiple brain regions encode sensory inputs, the current task, and choices. It remains unclear how these signals evolve. We simultaneously recorded neuronal activity from six cortical regions [middle temporal area (MT), visual area four (V4), inferior temporal cortex (IT), lateral intraparietal area (LIP), prefrontal cortex (PFC), and frontal eye fields (FEF)] of monkeys reporting the color or motion of stimuli. After a transient bottom-up sweep, there was a top-down flow of sustained task information from frontoparietal to visual cortex. Sensory information flowed from visual to parietal and prefrontal cortex. Choice signals developed simultaneously in frontoparietal regions and travelled to FEF and sensory cortex. This suggests that flexible sensorimotor choices emerge in a frontoparietal network from the integration of opposite flows of sensory and task information.

Our reactions are not always the same to the same sensory input. Depending on context, we can map the same input onto different actions. This involves a distributed network of brain regions. During visuomotor decisions, choice predictive activity has been found in frontoparietal regions, including the lateral intraparietal area (LIP) (14), prefrontal cortex (PFC) (1, 59), frontal eye fields (FEF) (7), and motor and sensory cortex (1013). However, it remains unclear how choice signals evolve. Do they flow bottom-up, flow top-down, or evolve concurrently across brain regions? Do choice signals in sensory regions reflect their causal effect on the decisions or feedback from decision stages (12)? Similarly, little is known about the flow of task signals. Neuronal activity encodes task rules in prefrontal (6, 8, 14, 15), parietal (2), and visual (16) cortices. Task-dependent attention modulates neuronal activity throughout sensory cortices (1719). It remains unknown how task signals evolve across these regions.

We trained two monkeys on a flexible visuomotor task (Fig. 1 and materials and methods). They categorized either the color (red versus green) or the direction (up versus down) of a colored visual motion stimulus, reporting it with a left or right saccade (Fig. 1A). A visual cue instructed animals about the task (motion or color, Fig. 1C). Each task was indicated by two different visual cues to dissociate cue and task-related activity. Color and motion spanned a broad range around the category boundaries (yellow and horizontal) (Fig. 1B and fig. S1). Both monkeys were proficient at categorizing the cued feature (Fig. 1D) (94% and 89% correct for motion and color tasks, respectively, excluding ambiguous trials with stimuli on the category boundary).

Fig. 1 Task, behavior, and neuronal information.

(A) Monkeys categorized the motion, direction, or color of centrally presented, colored random dot stimuli. Before stimulus onset, a central cue indicated which feature to categorize. Monkeys indicated their choice with a leftward or rightward saccade and held central fixation throughout each trial until their response. Monkeys were free to respond any time up to 3 s past stimulus onset. (B) Stimuli systematically covered motion, direction, and color space between opposite motion directions (up and down) and opposite colors (redand green; lab space). All stimuli were 100% coherent, iso-speed, iso-luminant, and iso-saturated. (C) Two different cue shapes cued each task. (D) Responses were strongly modulated by motion and color for the motion and color task, respectively. (E) Time courses of neuronal information in spiking activity about five different task variables averaged across all units and brain regions. Information is measured as percent variance of spiking explained by the variable of interest, independent of all other variables (%EV). (F) Percentage of units per region significantly encoding each type of information (P < 0.05). Dashed lines indicate chance level. (G) Average information encoded for each region and type of information. (H) Schematic display of the recorded brain regions. lPFC, lateral prefrontal cortex. (I) Time course of average motion, color, and choice information analyzed separately for motion and color categorization tasks. Information is log-scaled to facilitate comparison between tasks. All error bars denote SEM.

We recorded multi-unit activity (MUA) from up to 108 electrodes simultaneously implanted in six cortical regions acutely each day (Fig. 1H and materials and methods): FEF (532), dorsolateral PFC (1020), LIP (807), IT (57), V4 (155), and MT (123) (total of 2694 multi-units). For each multi-unit, we quantified how neural activity encoded cue identity, task (motion versus color), stimulus motion direction, stimulus color, and motor choice. Information was quantified as spiking variance across trials explained by each factor. All five types of information were quantified independently; for example, choice measured only information about the choice that was not explained by cue, task, color, or motion (see materials and methods). To rule out activity due to the saccade itself, we included neuronal activity up to 5 ms before saccade onset.

Averaging across all units revealed temporal dynamics of information (Fig. 1E). Cue information peaked directly after cue onset and stayed tonically elevated during cue presentation (latency to reach half maximum: 74 ± 1 ms SE). Task information showed a bimodal dynamic. A transient peak shortly after cue onset had a similar latency as cue information (100 ± 25 ms). This transient peak was followed by a dip and later rise of sustained task information (333 ± 15 ms). In contrast to cue information, task information increased during stimulus presentation. Motion and color information rose after stimulus onset with a significantly shorter latency for color (98 ± 2 ms) as compared with motion (108 ± 2 ms) information (P < 0.001). Last, choice information rose (193 ± 1 ms) before the motor responses (270 ms ± 3 ms) and significantly later than motion and color information (both P < 0.0001).

We quantified for each type of information the percentage of units with significant effects (Fig. 1F) and the average amount of information (Fig. 1G). We used the second half of the cue interval (0.5 to 1 s) for cue and task information, the interval from stimulus onset to the average response latency (1 to 1.270 s) for motion and color information, and the 200-ms interval preceding the saccade for choice information. We found significant encoding of each type of information in each region (P < 0.05 for all regions and information), but the regional profiles differed. In accordance with shape selectivity of V4 and IT, we found the most frequent and strongest cue information there. Task selectivity was frequent in all regions and strongest in V4 and IT. Motion and color information were strongest in MT and V4, respectively. Choice information was most frequent and strongest in LIP, FEF, and PFC. Task (motion versus color) had little effect on strength and dynamics of motion, color, and choice information (Fig. 1I) (20, 21). There was no evidence that only task-relevant sensory information was routed to frontoparietal stages and no evidence that choice information was present only in the task-relevant sensory region. In sum, all types of information were encoded across the entire visuomotor pathway, albeit with different incidences and strength.

Next, we investigated the temporal dynamics of information across regions. Cue information flowed bottom-up, rising first in MT, followed by LIP, V4, IT, FEF, and PFC (Fig. 2A). Most of the pairwise comparisons revealed significant latency differences between regions (Fig. 2B, P < 0.001). Task information showed very different dynamics (Fig. 2C). There was a significant early transient peak of task information (<150 ms) in IT and V4 only, without a latency difference between V4 and IT (P > 0.05). The latency of this peak in IT (72 ms) was not different (P > 0.05) from the latency of cue information in IT (also 72 ms). In V4, the transient peak of task information was slightly later (96 ms) than cue information (73 ms) (P < 0.05). Directly after this transient peak, task information was low in the PFC, but then appeared there first and flowed from PFC to LIP, MT, FEF, V4, and IT. Many pairwise latency comparisons were significant according to this pattern (Fig. 2D, P < 0.01). In particular, task information rose earlier in PFC and LIP than in FEF, V4, and IT (all P < 0.01). In summary, IT and V4 first extracted task information from the cues along with the encoding of cue identity. After this transient burst, there was a flow of sustained task information from PFC and LIP across the entire sensorimotor hierarchy.

Fig. 2 Dynamics of cue and task information.

(A) Each row displays for one brain region the average time course of neuronal information about cue identity. Left graphs display raw information (% EV, same scale for all regions). To support comparison across regions, right graphs display time courses normalized by maximum information for the interval of interest. The bottom right graph shows an overlay of all regions’ information time courses. Cue and stimulus onsets are at time = 0 s and time = 1 s, respectively. (B) Comparison of cue information latencies between regions. Latencies are quantified as the time to reach half maximum information. Black dots in the right graph indicate significant latency differences between regions. (C) Time courses of task information across regions. Same conventions as in (A). (D) Comparison of task information latencies between regions. Latencies were separately analyzed for the early transient peak around 100 ms and for the later sustained increase of task information after 200 ms. Early peak latencies were only estimated for regions that showed a significant effect (V4 and IT, P < 0.05). Same conventions as in (B). All error bars denote SEM.

Motion information rose first in MT, followed by LIP, V4, IT, FEF, and PFC (Fig. 3A). Color information rose first in MT, followed by V4, LIP, FEF, IT, and PFC (Fig. 3C). Most pairwise comparisons revealed latency differences between regions according to these sequences (Fig. 3, B and D, P < 0.05). Furthermore, color information appeared significantly earlier than motion information in V4, MT, PFC, and FEF (all P < 0.001). Analyzing motion and color tasks individually confirmed these results and showed that neuronal latencies for motion and color information were almost identical for both tasks (fig. S2).

Fig. 3 Dynamics of motion and color information.

Time courses and latencies of neuronal information about (A and B) motion direction and (C and D) color of the categorized stimulus. Stimulus onset is at time = 1 s. All other conventions as in Fig. 2.

Choice signals had a different dynamic. If spontaneous fluctuations of activity influenced animals’ choices, activity would predict the choice even before presentation of the motion-color stimulus. Indeed, for all regions except IT, significant choice information preceded stimulus onset (–0.5 to 1 s, P < 0.01). We ruled out that this prestimulus choice information merely reflected an effect of the previous trial (see materials and methods). We next investigated the build-up of choice information during decisions (Fig. 4). Because this reflects the forthcoming behavioral response, we time-locked analysis to the saccade. Choice information increased in LIP and PFC before FEF (Fig. 4B, P < 0.05), but there was no latency difference between LIP and PFC. Choice information increased later in V4 and MT than in LIP and PFC (Fig. 4B, all P < 0.05), suggesting feedback of choices from frontoparietal stages. Analyzing choice information for motion and color tasks individually confirmed the above results (fig. S3).

Fig. 4 Dynamics of choice information.

Response-locked (A) time courses and (B) latencies of neuronal information about the animals’ choice. Responses are at time = 0 s. Latency was not estimated for IT because there was no significant increase of choice information in IT in the analyzed interval (linear regression, P > 0.05). All other conventions as in Figs. 2 and 3.

Our results provide insights into the neuronal mechanisms underlying sensorimotor choices (summarized in fig. S4). First, sensory (cue, motion, or color), cognitive (task), and behavioral (choice) information was not confined to specific cortical regions but instead broadly distributed. This is incompatible with models of compartmentalized cortical function. Our results instead suggest a graded functional specialization of cortical regions with information shared between regions (22). Second, sensory information flowed feed-forward from sensory cortex. Third, task information was first extracted in an early, transient burst in higher sensory cortex (V4 and IT). This early transient may reflect the learned cue associations, that is, the grouping of the two cues for each task into one representation that is then fed forward to PFC and LIP. After the early transient, sustained task information appeared first in PFC and LIP and then spread to other regions. Thus, task information may need to reach PFC and LIP before being broadcast across the sensorimotor pathway (23). Fourth, choice predictive activity was present in sensory (V4 and MT), frontoparietal (LIP and PFC), and premotor (FEF) cortex before onset of the decision process. This suggests a link between spontaneous fluctuations of neuronal activity along the entire sensorimotor pathway and subsequent decisions. Fifth, choice signals first and simultaneously built up in PFC and LIP and then followed in FEF. Our findings accord with previous reports of ramping choice predictive activity in LIP (3), PFC (7), and FEF (7) but shed light on how choices are made in this network. Our results suggest that, although sensory information reaches LIP and FEF before PFC, the accumulation of sensory evidence occurs first and jointly in LIP and PFC before decision signals are relayed to FEF. Similar dynamics in PFC and LIP could indicate that accumulation of sensory evidence depends on their recurrent interactions (24, 25). The delayed choice signals in FEF may reflect the transformation of accumulated evidence into a discrete choice (26). Sixth, we found an increase of choice signals in LIP and PFC before MT and V4. This is consistent with feedback of choice signals from frontoparietal to sensory cortex (12, 13, 27). This may support cooperative computations between different hierarchical stages (28) and perceptual stability (27). In sum, flexible sensorimotor decisions are not a simple feed-forward process but result from complex temporal dynamics, including feed-forward and feedback interactions between frontal and posterior cortex.

Supplementary Materials

Materials and Methods

Figs. S1 to S4

References (2935)

References and Notes

  1. Acknowledgments: We thank J. Roy, C. von Nicolai, and J. Hipp for helpful discussions. This work was supported by National Institute for Mental Health (NIMH) grant 5R37MH087027 (E.K.M.), MIT Picower Innovation Fund (E.K.M.), National Institutes of Health (NIH) grant R00 MH092715 (T.J.B.), and the Centre for Integrative Neuroscience (Deutsche Forschungsgemeinschaft, EXC 307) (M.S.). All behavioral and electrophysiological data are archived at the Centre for Integrative Neuroscience, University of Tübingen, Germany.
View Abstract


Navigate This Article