Report

A memory of errors in sensorimotor learning

See allHide authors and affiliations

Science  12 Sep 2014:
Vol. 345, Issue 6202, pp. 1349-1353
DOI: 10.1126/science.1253138

Practice makes perfect — or does it?

How do we learn from past errors? Herzfeld et al. found that when we practice a movement, the human brain has a memory for errors that is then used to learn faster in new conditions. This memory for error exists in parallel with motor memory's two traditional forms: memory of actions and memory of external perturbations. They also proposed a mathematical model for learning from errors. This model explained previous experimental results and predicted other major findings that they later verified experimentally.

Science, this issue p. 1349

Abstract

The current view of motor learning suggests that when we revisit a task, the brain recalls the motor commands it previously learned. In this view, motor memory is a memory of motor commands, acquired through trial-and-error and reinforcement. Here we show that the brain controls how much it is willing to learn from the current error through a principled mechanism that depends on the history of past errors. This suggests that the brain stores a previously unknown form of memory, a memory of errors. A mathematical formulation of this idea provides insights into a host of puzzling experimental data, including savings and meta-learning, demonstrating that when we are better at a motor task, it is partly because the brain recognizes the errors it experienced before.

How does the brain alter behavior after experiencing an error? Classic theories assumed that the brain learns some fraction of the error regardless of its history or magnitude (1, 2). However, recent experiments (36) demonstrate that the brain learns relatively more from small errors than large errors, and can modulate its error sensitivity (79).

Understanding error sensitivity is important because it may provide insight into the phenomena of “savings” and “meta-learning.” Savings refers to the observation that when a person adapts to perturbation (A), and then the perturbation is removed (i.e., washout), they exhibit faster readaptation to (A) (10). Remarkably, savings of (A) is present even when washout is followed by adaptation to (-A), a perturbation in the opposite direction (11, 12). Current error-dependent models of learning cannot account for these observations (13, 14), nor explain meta-learning, where prior exposure to a random perturbation produces savings (15, 16).

We begin with a standard model of motor learning (1720) in which on trial Embedded Image, a perturbation Embedded Image is imposed on action Embedded Image so that the sensory consequences are Embedded Image. Based on their belief about the environment Embedded Image, the learner predicts the sensory consequences Embedded Image, and then updates his or her belief from the prediction error Embedded Image. Such learning typically depends on a decay factor Embedded Image, and error sensitivity Embedded Image

Embedded Image (1)

Consider an environment in which the perturbations persist from trial to trial, and another environment in which the perturbations switch (Fig. 1A). In a slowly switching environment, the brain should learn from error because the perturbations are likely to persist (learning from error in one trial will improve performance on the subsequent trial). However, in a rapidly switching environment, the brain should suppress learning from error because any learning will be detrimental to performance on subsequent trials.

Fig. 1 History of error alters error sensitivity.

(A) Reaching paradigm with force-field perturbations. The yellow circles note a perturbations state, and Embedded Image indicates probability of remaining in that state. The slow, medium, and rapidly switching environments are shown. One group of volunteers was trained in each environment. We measured error sensitivity through probe trials in which participants experienced a constant perturbation, sandwiched between two error-clamp trials. (B) Movement trajectories in the perturbation trial of the probe trials. Trajectories were averaged over five successive presentations of the probe. The errors in probe trials did not differ between groups. (C) Learning from error in the probe trials, measured as the change in force from the trial prior to the trial after the perturbation. (D) Learning from error in the probe trials, plotted as a percentage of the ideal force (left). Error sensitivity Embedded Image was measured as the trial-to-trial change in the percentage of ideal force divided by error (right). (E) Change in error sensitivity between the baseline block and the last five error-clamp triplets. Data are mean ± SEM.

Three groups of volunteers (n = 9 per group) made reaching movements while experiencing force perturbations from either a slow, medium, or rapidly switching environment (Fig. 1A). The mean of the perturbations was zero for all blocks (consisting of 30 trials). We measured error in a given trial and then computed the amount that was learned from that error (probe trials, purple bars, Fig. 1A). To quantify learning from error on trial Embedded Image, we measured the change in force from the trial before to the trial after the perturbation, Embedded Image (Fig. 1C). In block 1, learning from error was similar in the three groups (P > 0.99), and in all probe trials the perturbation produced similar errors [Fig. 1B; repeated measure–analysis of variance (RM-ANOVA), effect of group P > 0.8, interaction, P > 0.7]. However, individuals who experienced the slowly switching environment increased their learning from error (Fig. 1C), whereas those who experienced the rapidly switching environment suppressed this learning.

We measured the force produced on a given trial and computed a coefficient representing percent ideal (Fig. 1D). RM-ANOVA indicated a significant block by group interaction (P < 0.05), suggesting that the history of perturbations altered the amount of learning from error. Posthoc tests showed that in the slowly switching environment, participants learned more from error than in the rapidly switching environment (P < 0.03). This change in error sensitivity developed gradually with training (Fig. 1D). The slowly switching environment induced an increase in error sensitivity (Fig. 1E; changes in sensitivity from the first half to second half of the experiment, ANOVA, P < 0.05).

Is control of error sensitivity local to the experienced errors? In experiment 2, participants performed rapid out-and-back movements for which no visual feedback was available during the outward part of the reach, with the aim of hitting a target at the turn-around point of their movement. An occasional perturbation altered the feedback regarding hand position at the turn-around point (Fig. 2A). We measured the relation between error Embedded Image and learning from error (change in reach extent).

Fig. 2 Error sensitivity is a local function of experienced errors.

(A) Paradigm with visuomotor gain perturbations. (B) Perturbation schedule. Dashed lines indicate changes in the statistics of the environment. (C) Error sensitivity averaged over all error sizes measured over each environment block. (D) Learning from error measured at various error sizes. (E) Error sensitivity as a function of error magnitude. (F) Probability of error.

Group 1 (n = 10) experienced a perturbation schedule that transitioned from slow, medium, to rapid switching (Fig. 2B), whereas group 2 (n = 10) experienced the reverse. In group 1, error sensitivity decreased, whereas in group 2, error sensitivity increased (Fig. 2C). We measured the mean error sensitivity in each environment, resulting in three measurements for each subject across the experiment. RM-ANOVA showed a significant main effect of group (P < 0.005) and block (P < 0.001) and group-by-block interaction (P < 0.001). As the statistics of the perturbation changed, so did the error sensitivity.

We measured learning from error as a function of error in each environment (Fig. 2D). A given error produced greater learning when that error was experienced in a slowly switching environment (Fig. 2D, red line) (RM-ANOVA main effects of error size P < 10−4, and environment P < 0.001, posthoc between slow versus medium or fast, P < 0.001). We quantified error sensitivity at each error size (Fig. 2E) and found that error sensitivity had not changed globally, but had changed predominantly for smaller error sizes. RM-ANOVA of the absolute sensitivities between 0.25 and 2 cm showed a significant main effect of environment (P < 10−4), as well as a significant environment by error size interaction (P < 0.05). We found a significant difference in error sensitivity across environments for an error size of 0.25 cm (P < 0.05), but no significant difference for an error size of 2 cm (P > 0.1). Interestingly, the small error sizes for which the participants had shown the largest change in error sensitivity were also the most frequent errors (Fig. 2F). This hinted that control of error sensitivity was a function of error.

Current models of sensorimotor learning assume that error sensitivity Embedded Image is independent of error Embedded Image. This is true for state-space models of learning (18, 2124), as well as Kalman filter models of learning (5, 2527). However, suppose that sensory prediction errors are encoded in the nervous system with a set of basis elements, where each basis element Embedded Image has a preferred error Embedded Image. Further, suppose that error sensitivity is determined by a population coding

Embedded Image (2)

On trial Embedded Image, the motor command Embedded Image produces an error Embedded Image, as illustrated in the top part of Fig. 3A. The nervous system learns from this error and produces motor command Embedded Image on the subsequent trial, resulting in Embedded Image. In a slowly switching environment (top part of Fig. 3A), Embedded Image has the same sign as Embedded Image. In this case, error sensitivity should increase around Embedded Image (Fig. 3B, red line). By contrast, in a rapidly switching environment (Fig. 3A, bottom), Embedded Image has a different sign than Embedded Image. In this case, error sensitivity should decrease

Fig. 3 Theoretical model and experiment 3.

(A) On trial Embedded Image, the motor command Embedded Image is generated, resulting in error Embedded Image. If the error in trial Embedded Image is of the same sign as Embedded Image, then error sensitivity should increase (top). However, if the error experienced in trial Embedded Image has a different sign than Embedded Image, then error sensitivity should decrease (bottom). (B) Learning from error after experience of two consecutive errors from (A). Error sensitivity around Embedded Image increases if Embedded Image and decreases otherwise. (C) Model performance for slow, medium, and rapidly switching environments (gray line represents Embedded Image). Learning from error (D) is increased in the slowly switching environment and decreased in the rapidly switching environment. (E) Experiment 3 perturbation protocol. (F) Single-trial learning from a +8 N perturbation and a –4 N perturbation in probes 1 and 2. Learning is increased for the –4 N perturbation, while simultaneously decreased for a +8 N perturbation. (G) Learning from error normalized by the perturbation magnitude (4 or 8 N) in the first trial of each repetition of the rapidly (blue) and slowly switching (red) environments. Learning increased in the slowly switching (4 N) environment but decreased when the perturbation was rapidly switching (8 N). Error bars are SEM.

Embedded Image (3)

In Eq. (3), Embedded Image, Embedded Image, and superscript Embedded Image is the transpose operator. This rule is similar to the RPROP algorithm, a heuristic for adjusting the learning rate of machines (28), but has the unique feature of assuming that error sensitivity is via population coding of the error space.

Equations 1 to 3 represent a learner that stores two kinds of memory: a memory of the state of environment (Embedded Image, Eq. 1) and a memory of errors (Embedded Image, Eq. 3). We simulated the model (Fig. 3C, gray line) and found that in the slowly switching environment, error sensitivity increased in the neighborhood of the experienced errors, whereas in the rapidly changing environment, error sensitivity decreased (Fig. 3D).

Our model made a critical prediction: If the brain controlled error sensitivity via memory of errors, then it should be possible to simultaneously increase sensitivity for one error, while decreasing it for another. In experiment 3, we considered an isometric task in which participants (n = 16) produced a force to match a target (16 N) in the face of a perturbation. The perturbations were designed so that, according to our model, individuals would increase their sensitivity to –4 N errors, while simultaneously decreasing their sensitivity to +8 N errors.

In the baseline block, we probed sensitivity to +8 N and –4 N perturbations (probe 1, Fig. 3E). The resulting learning from error is plotted in Fig. 3F (probe 1). At baseline, participants responded to the +8 N and –4 N perturbations by learning a fraction of each error (Fig. 3F). We next produced 20 repetitions of a rapidly switching environment in which the perturbations were ±8 N (Fig. 3E, inset). After a period of washout, we then produced 15 repetitions of a slowly switching environment in which the perturbations were 0 N or +4 N. The critical aspect of our design was that the participants were never exposed to a –4 N perturbation. They nevertheless experienced –4 N errors (because removal of a learned +4 N perturbation results in a –4 N error).

The 8 N environment induced a decrease in sensitivity to a +8 N error, and subsequent exposure to the +4 N environment resulted in an increase in sensitivity to a –4 N error [Fig. 3G; RM-ANOVA showed a significant main effect of perturbation (P < 0.03) as well as a perturbation by block interaction, P < 0.01]. The critical question, however, was whether both of these changes in sensitivity were simultaneously present. After the slowly switching block of perturbations, we again probed sensitivity to +8 N and –4 N errors (probe 2, Fig. 3E). Compared to the baseline block (probe 1), learning from a +8 N error had decreased (P < 0.005), while simultaneously, learning from a –4 N error had increased (P < 0.05) (Fig. 3F). When we ran our model on the same sequence of errors that participants had experienced, the change in error sensitivity predicted by the model was highly correlated with the change observed in our participants (R2 = 0.65; P < 10−8, fig. S1), suggesting that history of error induced changes in error sensitivity in the region of the experienced errors.

This new model of learning provided insights on a wide range of puzzling experiments, including the phenomena of savings and meta-learning (fig. S2). It predicted that when one is better at a task than before, it is not because the brain recalled the motor commands, but because it recognized the errors—the errors for which error sensitivity had been altered. In addition, the model predicted that savings and meta-learning could be blocked by controlling the errors that are experienced during learning.

In experiment 4, volunteers participated in a visuomotor rotation experiment (Fig. 4A and fig. S4, n = 10 per group). The control group (ANA) experienced a +30° perturbation followed by extended washout and then relearning of +30°, a protocol that should produce savings (13). According to our model, savings occurs because during the initial exposure to (A), the stable sequence of perturbations increase error sensitivity, and these errors are revisited in the subsequent test of (A). If so, we should be able to block savings by presenting (A) gradually (GNA group, fig. S4), preventing prior exposure to errors that are visited at the onset of (A).

Fig. 4 Saving occurs only when previously experienced errors are revisited.

(A) A visuomotor perturbation experiment. Gray arrows indicate 1- to 2-min set breaks. (B) Performance in the final +30° perturbation. ANA and BNA groups show savings, i.e., faster learning of the perturbation compared to control (naïve). Exponential fits are shown for the group data. (C) The BwaitNA group does not exhibit savings. (D) Exponential time constants are compared to controls (*P < 0.05, **P < 0.01). A smaller time constant indicates faster learning. (E) Comparison of the errors (i.e., aftereffects) experienced by the BNA and BwaitNA groups. The BwaitNA group experienced smaller errors owing to the presence of the set break.

Furthermore, we should be able to produce savings in a very different way: Expose participants to perturbation (B) and then present sudden washout (Fig. 4A, BNA). During washout, they are exposed to a sequence of stable errors, which increase error sensitivity for those errors. Notably, the washout-induced aftereffects are errors that are also experienced during subsequent test of (A). If the meta-learning in BNA is due to errors that are experienced during washout of (B), we should be able to eliminate it by reducing the washout-induced errors. In BwaitNA, a wait period was inserted between –30° training and washout, reducing the size of aftereffects associated with the transition from –30° to washout (Fig. 4E). We also tested this idea in a different way: gradual washout (B) (group BGNA, fig. S4). In summary, the model predicted savings in ANA but not GNA, and meta-learning in BNA but not BwaitNA and BGNA. Our experimental results confirmed these predictions (Fig. 4, C and D, and fig. S4).

We found that during learning, the brain controlled error sensitivity in a principled way: learning more from error when perturbations were likely to persist, and less when perturbations were likely to change. Error-sensitivity modulation was specific to the experienced errors, suggesting that training produced a memory of errors. This idea accounted for a host of puzzling observations, including saturation of error sensitivity (5, 6, 29), the phenomenon of meta-learning (16), examples of savings (1012), and reinforced repetition (15).

The model predicted that meta-learning vanishes when a small delay or gradual washout alters the history of errors (Fig. 4A), demonstrating that savings depends crucially on the memory of errors that is accumulated during training. This memory of errors likely exists in parallel with the two traditional forms of motor memory, memory of perturbations (23) and memory of actions (30).

In our model, we chose to describe the learner as a process with a single time scale. However, data suggest that learning from error depends on a fast and a slow process with different error sensitivities (23, 25, 31). We speculate that the memory of errors exerts its influence through the error sensitivity of the fast process, and its manipulation through history of errors may be a useful strategy to speed recovery during rehabilitation (32).

Supplementary Materials

www.sciencemag.org/content/345/6202/1349/suppl/DC1

Supplementary Text

Figs. S1 to S4

References (3340)

References and Notes

  1. Acknowledgments: This work was supported by NIH grants T32EB003383, T32GM007057, R01NS078311, and 1F31NS079121. We thank J. Krakauer, J.-J. Orban de Xivry, and A. A. Ahmed for comments. D.J.H., P.A.V., M.K.M., and R.S. conceived and designed the experiments. D.J.H. performed the experiments, analyzed the data, and performed the simulations. R.S. and D.J.H. wrote the paper. The data reported in this paper and in the supplementary materials are available from the corresponding author.
View Abstract

Navigate This Article