Cognitive Inflexibility After Prefrontal Serotonin Depletion

See allHide authors and affiliations

Science  07 May 2004:
Vol. 304, Issue 5672, pp. 878-880
DOI: 10.1126/science.1094987


Serotonergic dysregulation within the prefrontal cortex (PFC) is implicated in many neuropsychiatric disorders, but the precise role of serotonin within the PFC is poorly understood. Using a serial discrimination reversal paradigm, we showed that upon reversal, selective serotonin depletion of the marmoset PFC produced perseverative responding to the previously rewarded stimulus without any significant effects on either retention of a discrimination learned preoperatively or acquisition of a novel discrimination postoperatively. These results highlight the importance of prefrontal serotonin in behavioral flexibility and are highly relevant to obsessive-compulsive disorder, schizophrenia, and the cognitive sequelae of drug abuse in which perseveration is prominent.

Serotonin [5-hydroxytryptamine (5-HT)] is implicated in cognition and impulsivity and is of particular relevance to our understanding of the psychopathology and treatment of psychiatric disorders such as depression, schizophrenia, and obsessive compulsive disorder (OCD). In all of these disorders, local abnormalities in prefrontal cortex (PFC) structure (1), neurochemistry (2), or activation (3) have been identified, and as is consistent with prefrontal dysfunction, cognitive inflexibility is a prominent feature. Moreover, drugs such as ecstasy (4) and amphetamine (5) have been shown to impair cortical 5-HT neurotransmission, but the functional consequences of such serotonin dysregulation are not known.

Two approaches used to study the effect of central 5-HT on behavior and cognition are dietary tryptophan depletion in humans (6) and destruction of the ascending serotonergic projections in animals, using the selective neurotoxin 5,7-dihydroxytryptamine (5,7-DHT) (7). However, neither approach can differentiate between the roles of 5-HT in distinct forebrain structures, although there is indirect evidence that not all forebrain structures are equally modulated by 5-HT. Tryptophan depletion in humans has little effect on the performance of tasks activating the dorsolateral region of the PFC, such as spatial working memory or planning (6). However, tryptophan depletion does impair performance on visual discrimination reversal tasks (6, 8), on which performance is disrupted by orbitofrontal cortex (OFC) lesions in nonhuman primates (9, 10) and humans (11). There are therefore similarities between tasks that depend on OFC function and tasks that are affected by tryptophan depletion, suggesting a role for 5-HT in processes mediated by the OFC.

The present study investigated the effects of selective prefrontal depletion of 5-HT in primates on an OFC-dependent task: visual discrimination reversal. In this task, marmosets performed two-choice visual discriminations. They were trained preoperatively on two discriminations (D1 and D2) (Fig. 1A) before receiving either multiple 5,7-DHT injections throughout the PFC or a sham control procedure. Postoperatively, they were retested on the retention of D2, on acquisition of a novel discrimination, and then on four reversals (12).

Fig. 1.

(A) Experimental protocol. Monkeys performed two-choice discriminations in which the choice of the correct stimulus (+) resulted in a 5-s banana milkshake reward. They performed 30 trials per session, one session per day, with the criterion for completion of a discrimination being 90% or more trials correct in one session. D1 and D2 were completed before surgery, and D2 retention (D2 ret) was tested after recovery from either a 5,7-DHT lesion or a sham control procedure. All monkeys then acquired D3 to criterion before receiving a series of reversals. Real stimuli were multicolored. (B) 5,7-DHT lesions (solid bars, n = 3 monkeys) induced a selective increase in the number of perseverative errors made across the series of reversals (Rev) in comparison to controls (open bars, n = 3 monkeys). Analysis of variance using the model group2 × (error type3 × reversal4 × subject) where “error type” = perseveration, chance, or learning, revealed a significant main effect of error type (F2,8 = 29.466, P < 0.001), reversal (F3,12 = 20.765, P < 0.001), and a three-way error type × reversal × group interaction (F6,24 = 2.813, P = 0.032). Simple interaction effects revealed a reversal × group interaction for perseverative errors (F3,12 = 6.695, P = 0.007), which simple main effects revealed to be due to an increase in perseverative errors in lesioned monkeys during reversals 2 (F1,5 = 30.640, P < 0.001, **) and 3 (F1,5 = 12.019, P = 0.026, *), but not during reversals 1 (F < 1) or 4 (F1,5 = 3.304, P = 0.143). There was no effect of the lesion on errors made during the chance or learning stages (largest F value = 2.218 for reversal 4 in chance stage; largest F value = 2.286 for reversal 3 in learning stage).

Preoperatively, marmosets destined to receive a 5,7-DHT lesion or sham surgery did not differ in the number of errors they made before the criterion for completion was reached on two novel visual discriminations. Postoperatively, the lesion had no effect on either the ability to remember a previously learned visual discrimination or the ability to acquire a new discrimination (Table 1). However, upon reversal of the reward contingency, lesioned monkeys made significantly more errors than did controls. Although control monkeys showed a steady decline in the number of errors made before criterion was reached from the first to the fourth reversal, lesioned monkeys did not (Table 1). Closer examination of the types of errors revealed that lesioned monkeys made significantly more perseverative errors than did controls (Fig. 1B; statistics provided in the legend). The finding that lesioned monkeys did not make more errors than controls during the chance and learning stages suggests a selective effect of prefrontal 5-HT depletion on perseverative responding.

Table 1

. Mean total error scores and SEM values (square root–transformed) for control and lesion groups at each experimental stage.

There was no evidence for any motoric or general motivational effects of the lesion, as shown by the comparable latencies of control and lesioned groups to make correct responses (F's < 1) or incorrect responses (F's < 1) and to obtain reward after a correct response (F's < 1).

In vivo microdialysis revealed that 5,7-DHT lesions resulted in substantial reductions in baseline and K+-evoked extracellular levels of 5-HT and its metabolite 5-hydroxyindoleacetic acid (5-HIAA) in the OFC measured 3 to 10 months after surgery (Fig. 2, A and B). Neurochemical quantification with high-performance liquid chromatography post mortem (4 to 12 months after surgery) confirmed the presence of substantial depletions of 5-HT within the PFC after 5,7-DHT lesions. Significant reductions in 5-HT were seen in the OFC, lateral PFC, dorsal PFC, and pregenual medial PFC (Table 2; statistics provided in the legend). There were also reductions in the neighboring primary motor/premotor cortex and the postgenual anterior cingulate cortex, but no alterations in motor ability, either within the home cage or when using the test apparatus, were observed. Effective 5-HT depletion was confirmed by significant depletions of the metabolite 5-HIAA in all of the above regions, except for the motor/premotor cortex (table S2). Midcingulate cortex 5-HT/5-HIAA levels were not significantly depleted, and pilot data from a unilaterally lesioned marmoset examined 2 weeks after surgery revealed no 5-HT/5-HIAA depletions in more distant cortical structures such as the anterior and posterior parietal cortex, nor in subcortical forebrain structures, including the dorsal and ventral striatum, hypothalamus, basal forebrain, temporal pole, and hippocampus (table S4). Administration of the reuptake blockers GBR-12909 and Nisoxetine successfully protected the dopaminergic and noradrenergic systems in the PFC, respectively, because no region showed significant alterations in dopamine (P > 0.05), the metabolite DOPAC (P > 0.05), or noradrenaline (P > 0.05). Pilot catecholamine data from a unilaterally lesioned marmoset at 3 months after surgery also revealed no substantial depletions at this earlier time point, eliminating the possibility of transient but behaviorally disruptive catecholamine depletion before the final histochemical analysis (table S3).

Fig. 2.

Mean extracellular 5-HT (A) and 5-HIAA (B) levels in the OFC in control monkeys (open squares) and 5,7-DHT–lesioned monkeys (solid squares). Values given are those of 5-HT and 5-HIAA for each successive 20-min dialysis sample, expressed in femtomoles. Samples 1 to 4 were taken during the baseline period, and the arrow indicates the local administration of K+ (75 mM for 10 min) at the start of sample 5. 5-HT was undetectable in lesioned monkeys, and the K+-evoked response was severely blunted (80.9% reduction). An equivalent pattern was seen for OFC 5-HIAA, because both baseline and K+-evoked 5-HIAA were severely reduced (99.1 and 97.3% of control levels, respectively).

Table 2.

Mean levels of 5-HT, dopamine, and noradrenaline (expressed as pmol/mg of wet tissue weight ± SEM) in the frontal cortex of the control and lesion groups, and the percentage depletion of serotonin (±SEM) in marmosets with 5,7-DHT lesions of the frontal cortex. The asterisk indicates that mean scores of lesioned animals differ significantly from those of the control group when analyzed by independent samples t test. OFC, orbitofrontal cortex (t4 = 2.878, P = 0.045); LAT, lateral granular PFC (t4 = 6.25, P = 0.003); MED, medial PFC (t4 = 4.85, P = 0.008); M/PM, primary motor and premotor cortex (t4 = 3.57, P = 0.023); DORSAL, dorsal granular cortex (t4 = 4.83, P = 0.008); C1, anterior cingulate cortex (t4 = 2.83, P = 0.048); C2, midcingulate cortex (P > 0.05). In all cases n = 3 monkeys, except for control C2 values, where n = 2 due to data loss.

5-HT (pmol/mg) Dopamine (pmol/mg) Noradrenaline (pmol/mg)
Control Lesion % depletion Control Lesion Control Lesion
OFC 0.7 ± 0.2 0.12 ± 0.05 82.5 ± 8* 1.8 ± 0.3 1.6 ± 0.4 3.5 ± 0.5 5.09 ± 0.68
LAT 0.6 ± 0.1 0.09 ± 0.03 85.5 ± 5* 1.3 ± 0.2 1.3 ± 0.1 4.4 ± 0.3 4.75 ± 0.93
MED 0.9 ± 0.2 0.12 ± 0.06 86.2 ± 7* 1.4 ± 0.3 1.7 ± 0.5 5.1 ± 0.2 4.41 ± 0.29
DORSAL 0.6 ± 0.1 0.17 ± 0.02 69.0 ± 4.3* 2.4 ± 0.1 2.7 ± 0.3 6.4 ± 1.1 6.75 ± 0.77
M/PM 0.7 ± 0.1 0.24 ± 0.04 64.0 ± 6.7* 2.7 ± 0.6 2.8 ± 0.5 7.6 ± 0.7 9.32 ± 0.55
C1 0.5 ± 0.2 0.13 ± 0.04 74.0 ± 8.9* 2.0 ± 0.6 2.3 ± 0.2 7.5 ± 2.6 8.43 ± 0.7
C2 0.33 ± 0.04 0.19 ± 0.04 35.3 ± 13.6 1.3 ± 0.3 1.6 ± 0.3 6.7 ± 0.86 9.40 ± 0.86

The present results demonstrate the deleterious behavioral effects of 5,7-DHT–induced selective prefrontal 5-HT depletion. Because reversal learning is sensitive to OFC damage in monkeys and humans (9, 11, 13), these data may suggest an important role for 5-HT in OFC function during performance of this task. Although contributions of 5-HT in other frontal areas cannot be ruled out, previous studies in humans have suggested that OFC function might depend heavily on 5-HT. Acute tryptophan depletion reduces central 5-HT and reproduces some of the deficits in decision-making produced by OFC lesions due to altered processing of reward cues (14). The present study provides a direct demonstration that 5,7-DHT–induced depletion of prefrontal, rather than whole-brain, 5-HT impairs reward-related processing. Although it cannot be ruled out that nonspecific effects of 5,7-DHT in the local lesion environment, such as reactive gliosis and changes in glutamate metabolism, may have behavioral consequences (15), it is unlikely that such effects contributed to the effects seen in the present study, given that these post-surgery effects are relatively short-lived and that similar effects produced by the catecholamine neurotoxin 6-hydroxydopamine (6-OHDA) have no effect on reversal learning, as discussed below.

These effects of 5-HT depletion contrast with previous studies investigating the contribution of other forebrain neuromodulator systems to reversal learning. In the marmoset, 6-OHDA–induced depletion of catecholamines in the PFC and dopamine in the caudate nucleus does not affect the ability to reverse stimulus-reward associations (16), although results from systemic drug studies are more ambiguous (17, 18). Comparable with the present results, cholinergic basal forebrain lesions in marmosets were found to impair reversal learning (19), partly because of increased perseveration (20). Thus, OFC function may be modulated by 5-HT and acetylcholine actions; the latter, in part, due possibly to interactions with 5-HT (21).

Extensive evidence implicates the OFC in reward-related information processing, including the association of a visual stimulus with reward or punishment (22, 23) and the use of reward information in guiding actions (22). Indeed, the OFC has been shown to respond to reward and punishment in humans within a reversal learning paradigm (24), and humans with OFC damage exhibit deficits in decision-making that are attributable to an inability to plan for future reinforcing consequences and to learn from feedback (25), including somatic markers. It is possible that 5-HT normally mediates some of this feedback.

The present results have implications for our understanding of a number of neuropsychiatric disorders. OCD sufferers show deficits in reversal learning that positively correlate with the severity of their OCD symptoms, and extensive evidence now implicates overactivity of the OFC in the pathophysiology of OCD (2, 26). OCD can be treated successfully with serotonergic drugs, including the tricyclic clomipramine and the selective serotonin reuptake inhibitor paroxetine (27), suggesting that inhibition of the 5-HT transporter is necessary for symptom amelioration. Schizophrenics are also impaired in reversal learning because of perseveration—a deficit particularly associated with negative symptomatology (28). An increase in negative symptom severity is associated with decreased OFC gray matter volume (1), and it has been suggested that OFC dysfunction plays a part in the manifestation of negative symptoms in schizophrenia. In addition, chronic cocaine treatment induces perseverative responding in discrimination reversal (29). The present results indicate that PFC 5-HT dysfunction (most likely OFC 5-HT dysfunction) may specifically contribute to such perseverative behavior.

Supporting Online Material

Materials and Methods

Tables S1 to S4


References and Notes

View Abstract

Stay Connected to Science

Navigate This Article