Integration of What and Where in the Primate Prefrontal Cortex

See allHide authors and affiliations

Science  02 May 1997:
Vol. 276, Issue 5313, pp. 821-824
DOI: 10.1126/science.276.5313.821


The visual system separates processing of an object’s form and color (“what”) from its spatial location (“where”). In order to direct action to objects, the identity and location of those objects must somehow be integrated. To examine whether this process occurs within the prefrontal (PF) cortex, the activity of 195 PF neurons was recorded during a task that engaged both what and where working memory. Some neurons showed either object-tuned (what) or location-tuned (where) delay activity. However, over half (52 percent, or 64/123) of the PF neurons with delay activity showed both what and where tuning. These neurons may contribute to the linking of object information with the spatial information needed to guide behavior.

Anatomical segregation of processing is an important principle of neural organization. Even within a modality, largely separate pathways process different attributes of the same stimulus. Perhaps the best explored example of segregation is in the visual system, where the analysis of visual scenes is carried out by at least two pathways. A “ventral pathway” through inferior temporal (IT) cortex processes information about features that identify objects, such as shape and color (object, or “what” information), and a “dorsal pathway” through posterior parietal (PP) cortex processes information about location and spatial relations among objects (spatial, or “where” information) (1). This example raises the question of where and how information about object identity is integrated with information about object location. One region that may play a role in integration is the prefrontal (PF) cortex, which receives inputs from virtually all of the brain’s sensory systems (2) and has long been thought to be an area where diverse signals are integrated to serve higher order cognitive functions.

A major contribution of the PF cortex to cognition is the active maintenance of behaviorally relevant information “online,” a process known as working memory (3). Working memory is typically studied in tasks in which an animal must remember a cue stimulus over a delay period and then make a behavioral response based on the cue. Physiological studies in monkeys have revealed that many PF neurons are highly active during the delay of such tasks (4). The activity is often cue-specific, suggesting that this “delay activity” is the neural correlate of the working memory trace. Given its central role in cognition, PF neurons that contribute to working memory are obvious candidates for integrating diverse signals. However, the extent to which different types of information, such as what and where, are integrated within the PF cortex is not well understood. Highly processed spatial information from the PP cortex and object information from the IT cortex are received by separate regions of the PF cortex, the dorsolateral (areas 46 and 9) and the ventrolateral (area 12) PF cortex, respectively (5), but there are interconnections between these regions that could bring what and where together (2, 6).

Physiological studies have found that different neurons and even different regions of the PF cortex convey either object information (in the ventrolateral PF cortex) or spatial information (in the dorsolateral PF cortex), but no neurons have been reported to convey both (7). In previous studies, however, working memory for what and where was examined in two separate tasks: an object task and a spatial task. This separation rarely occurs in the real world and it raises the possibility that the apparent segregation of what and where working memory reflected an artificial behavioral segregation. Thus, to investigate whether object and spatial information is integrated by individual PF neurons, we employed a task in which what and where are used together.

On each trial (Fig. 1), while the monkey maintained fixation of a fixation spot, a sample object was briefly presented at the center of gaze. After a delay, two test objects were briefly presented at two of four possible extrafoveal locations. One of the test objects matched the sample, the other was a nonmatch. After another delay, the monkey had to make a saccade to the remembered location of the match. Thus, this task required that the monkey, within a trial, link what with where. It had to remember the object’s identity over the first delay (the what delay), use that information to find the match, and then remember its location over the second delay (the where delay) (8).

Figure 1

A typical behavioral trial. The correct behavioral response, a saccadic eye movement to the remembered location of the matching object, is indicated by the arrow. The order of presentation is from the upper left to the lower right.

We recorded the activity of 195 neurons from the lateral PF cortex of two monkeys (9). Many of the neurons were activated during the delay intervals. To discern whether the level of delay activity was related to the information retained in memory, we performed analyses of variance (ANOVAs) on each neuron separately (10). The sample object was the factor for a one-way ANOVA on activity from the what delay (OBJECT factor). The cued location (LOCATION factor) and object used to cue it (OBJECT factor) were used for a two-way ANOVA applied to the where delay activity. On the basis of the ANOVAs (evaluated atP < 0.01), 64% of the neurons (123/195) showed delay activity that varied depending on either the object or location, or both.

Some PF neurons (8/123, or 7%) showed delay activity that was significantly tuned to the sample object only. During the what delay while the monkey viewed a blank screen and held the sample object in working memory, these neurons were highly active, with different sample objects evoking different levels of activity (OBJECT factor,P < 0.01). By contrast, during the where delay while the monkey had to hold location information in working memory, none of these cells were selective for the cued location (LOCATION factor,P > 0.01) or for the sample object (OBJECT factor,P > 0.01). For example, the neuron shown in Fig.2A showed significant sample object–tuned activity during the what delay. During the where delay when the task demands shifted to retaining location information, the neuron’s activity decreased relative to its level of activity during the what delay. We termed these neurons “what” cells. They appear to be specialized for object working memory.

Figure 2

Responses of single PF neurons showing either object-tuned (A) or location-tuned (B) delay activity. The small horizontal line on the left of each histogram indicates the time of the sample object presentation, and the line in the middle indicates presentation of the test objects. “Good object” and “poor object” refer to the objects used as samples. “Good location” and “poor location” refer to the locations cued by the matching object. “Good” or “poor” refer to the object or location that elicited the most or least activity, respectively. Bin width, 20 ms.

Other neurons (51/123, or 41%) were not selective for the sample object during the what delay (OBJECT factor, P > 0.01) but showed significant tuning for locations during the where delay (LOCATION factor, P < 0.01). For nearly all of these neurons (48/51, or 94%), object information had no effect on the where delay activity (OBJECT factor, P > 0.01, LOCATION X OBJECT interaction, P > 0.01). For example, the neuron shown in Fig. 2B exhibited relatively little activity until a location was cued by the matching object. Then, during the where delay, this neuron was highly active, with different locations eliciting significantly different levels of activity. We termed these neurons “where” cells because they appear to be specialized for spatial working memory (11).

Over half of the PF neurons with delay activity (64/123, or 52%) were not specialized but rather appeared to contribute to both object and spatial working memory. Their what delay activity varied significantly with the object used as the sample (OBJECT factor, P < 0.01), and their where delay activity varied significantly with which location was cued (LOCATION factor, P < 0.01) (Fig.3A). They were highly selective for both objects and locations. On average, there was a 64% increase in what delay activity after a good (preferred) sample object over the activity after a poor (nonpreferred) sample object and a 71% increase in where delay activity after cueing of a good location over the activity after a poor location. Thus, these “what-and-where” cells conveyed object and spatial information during different epochs of the same behavioral trial and appeared to contribute to both object and spatial working memory. What cells, where cells, and what-and-where cells were distributed equally between the dorsolateral PF cortex and the ventrolateral PF cortex (12).

Figure 3

(A) Response of a single PF neuron showing object-tuned activity in the what delay and location-tuned activity in the where delay. (B) Object and location information. Average histogram of the 28 what-and-where cells for which both location and object significantly affected where delay activity (on the basis of an ANOVA; see text). Cueing a good location with a good object elicited more activity than cueing a good location with a poor object. However, a poor location elicited less activity than a good location, regardless of which object cued it. See Fig. 2for conventions.

Because we cued each location with an object, location-tuned activity could have conveyed either location information alone or information about both the location and the matching object that cued it. We found examples of both. The two-way ANOVA revealed that a little over half of the what-and-where cells (36/64, or 57%) showed location tuning in the where delay that was unaffected by the match object (LOCATION factor,P < 0.01, OBJECT factor, P > 0.01; LOCATION X OBJECT interaction, P > 0.01); that is, the level of activity for a given location was the same regardless of which object cued it. We also conducted, for each neuron, a discriminant analysis on the activity from the where delay to measure the amount of information carried about the cued location and the matching object. The discriminant analysis attempted to classify, on the basis of a neuron’s firing rate on each trial, which one of the four locations was cued or which one of the four objects was the match (13). Because four objects and four locations were used, chance performance for each classification was 25%. For these neurons, the mean successful classification rate for locations on the basis of where delay activity was 33.3%, which was significantly greater than chance (t test, P < 0.001) (14). By contrast, the mean classification rate for objects on the basis of where delay activity, 25.6%, was not significantly different from chance (P = 0.206). Thus, after having conveyed object information in the what delay, these neurons “switched modes” and conveyed only location information in the where delay. This transformation mirrors the demands of the behavioral task.

For the remaining what-and-where cells (28/64, or 44%), both object and location information significantly affected the where delay activity (LOCATION factor, P < 0.01; OBJECT factor,P < 0.01, or LOCATION X OBJECT interaction,P < 0.01). The predominantly location-tuned activity was further modulated by the object that cued the location. For a given location, the where delay activity was higher if a good object cued it than if a poor object cued it (Fig. 3B). For these neurons, the mean classification rate for locations on the basis of where delay activity was 34.2%, whereas the classification rate for objects, 28.3%, was smaller, but significantly above chance (both different from chance,P < 0.001) (15). Thus, the where delay activity of these neurons reflected both the cued location and the object that appeared in it, that is, integrated what and where information.

Recent studies have emphasized a segregation of object and spatial information processing in both the visual cortex and the prefrontal cortex. The results of the present study indicate that when object and location information are used together (as is typically the case in the real world), information about these attributes converges in the PF cortex. Indeed, the results support the notion that a function of the PF cortex is to integrate disparate information (16). What and where signals could be integrated through interconnections between dorsolateral and ventrolateral PF cortices (2, 6), through converging projections from the parietal and temporal cortex on the frontal cortex (17), through cross-talk in the visual cortex (18), or through a combination of these pathways. In any case, single PF neurons that process both what and where signals may contribute to the linking together of object information with the spatial information needed to direct action. They may also help synthesize a unified representation of objects in their places. Indeed, the activity of many neurons simultaneously reflected a location and the object that appeared in it. They may play a role in integrating what and where in working memory.

Finally, the fact that the properties of many of the delay neurons mirrored the requirements of the task (they conveyed first object, then location, information) suggests that the PF cortex is “tuned” by behavioral demands (19). Functional topography of sensory cortical areas changes with experience (20). It may be that the PF cortex, which plays a central role in the flexible guidance of behavior, exhibits extensive functional plasticity. Thus, the PF cortex may be highly modifiable, its representations changing to meet the demands placed on it.

  • * To whom correspondence should be addressed. E-mail: ekm{at}


Stay Connected to Science

Navigate This Article