Technical Comments

Comment on “Multiyear Prediction of Monthly Mean Atlantic Meridional Overturning Circulation at 26.5°N”

Science  02 Nov 2012:
Vol. 338, Issue 6107, pp. 604
DOI: 10.1126/science.1222566


Matei et al. (Reports, 6 January 2012, p. 76) claim to show skillful multiyear predictions of the Atlantic Meridional Overturning Circulation (AMOC). However, these claims are not justified, primarily because the predictions of AMOC transport do not outperform simple reference forecasts based on climatological annual cycles. Accordingly, there is no justification for the “confident” prediction of a stable AMOC through 2014.

Matei et al. (1) claimed that an initialized global climate model (GCM) could produce skillful multiyear forecasts of the Atlantic Meridional Overturning Circulation (AMOC) strength at 26.5°N over the time period 2004 to 2008. We show that their statistical evaluation of forecast skill is not robust, resulting largely from the limited forecast evaluation period and the use of inappropriate reference forecasts. Due to these shortcomings, we argue that the claim in (1) of meaningful forecast skill as a basis for making future predictions of AMOC variability is unjustified.

Since 2004, a unique and exciting observing system has been providing records of the strength of the AMOC at 26.5°N Rapid Climate Change-Meridional Overturning Circulation and Heatflux Array (RAPID-MOCHA) (2, 3). Matei et al. use data from RAPID-MOCHA to assess the ability of a GCM (4) to predict the evolution of the AMOC strength after the GCM’s oceanic state is “spun up” with estimates of the observed momentum and heat fluxes between the atmosphere and ocean, in an attempt to capture the initial state of the climate system at a given time (these forecasts are referred to as “initialized”).

As a skill metric, Matei et al. use the correlation between the observed and predicted monthly AMOC strength over a year and show this metric at different leads (1 to 4 years) from four starting dates (2004 through 2007). A critical element of their analysis is the inclusion of the annual cycle in both the observations and the predictions. This is crucial because over the time period 2004 to 2008, the observed and predicted AMOC is dominated by a strong annual cycle, which can inflate the apparent performance of the prediction system without any practical benefit. For example, a useful forecast of monthly mean temperatures in Europe does not consist of determining whether it will be warmer in August than February but whether it will be warmer or cooler than what we expect for a typical August.

Therefore, our baseline knowledge of the climatological mean annual cycle must be taken into account in assessing the skill of the forecasts. A typical method for removing the artificial inflation of skill from the annual cycle is to subtract a repeating climatology from observations and predictions or to work with quantities that have been averaged over a year or multiple years. Alternatively, should the mean annual cycle be retained, as was done in the Matei et al. analysis, we argue that a proper metric of forecast performance involves comparing the skill of the initialized forecasts against that from a simple repeating climatological annual cycle.

Figure 1 [adapted from figure 2 of (1)] compares the skill of the Matei et al. initialized forecasts of AMOC transport with the skill obtained from simply assuming a repeating climatological annual cycle. Two slightly different AMOC climatologies, both based on “uninitialized” GCM integrations (i.e., with initial conditions not constrained by observations), were used for this baseline (or “null”) forecasting strategy: (i) an average over 1860 to 2000 from the GCM simulations of (4) and (ii) an average over 1948 to 2000 using the GCM of (1). For only 3 of 13 cases do the initialized forecasts have nominally larger correlation scores than the climatological null forecasts made using (4) (GCM climatology). Similarly, initialized forecasts outperform the Matei et al. climatology in only 5 of the 13 cases. Thus, there is no evidence in (1) of skill from these initialized AMOC forecasts.

Fig. 1

Skill of initialized forecasts of AMOC strength from (1) against two repeating climatologies [figure adapted from figure 2 in (1)]. Black, violet, blue, and yellow solid lines and symbols indicate the 12-month correlation of forecasts initialized in 2004, 2005, 2006, and 2007, respectively, over the first, second, third, and fourth year of the forecast. Circles connected by a dashed line indicate the 12-month correlation of the monthly climatology of AMOC strength at 26.5°N based on the repeating climatology of uninitialized experiments with the Fifth Generation of the Max Planck Institute Atmospheric GCM (MPI-ECHAM5) model (4), with color coding consistent with the solid lines. The left panel shows results using the 1861 to 2000 climatology from the historical radiative forcing (20c3m) MPI-ECHAM5 model run submitted to the Third Coupled Model Intercomparison Project (CMIP3) (13); the right panel is based on the 1948 to 2000 uninitialized MPI-ECHAM5 (4) used in (1). The initialized forecast outperforms a repeating climatology if, for a given lead time, a symbol connected to a solid line has a higher correlation than the symbol connected to the dashed line of the same color. The 1860 to 2000 climatology (left panel) outperforms the initialized forecasts except for 2004 at Lead 1 (first black symbol, which was a year with only 9 months of observations), 2007 at Lead 3, and 2008 at Lead 4 (third and fourth violet symbols). Meanwhile, the 1948 to 2000 climatology (right panel) outperforms the initialized forecasts except for 2004 at Lead 1 (first black symbol), 2005 at Lead 1, 2007 at Lead 3, and 2008 at Lead 4 (first, third, and fourth violet symbols) and 2008 at Lead 3 (third blue symbol). Horizontal black dashed line indicates the P = 0.1 confidence level on correlation (14). [The correlations of the uninitialized climatologies to observations shown in both panels were computed by the authors of (1) using data of (1, 4) and provided to us in the process of preparing this Comment; the correlations from the initialized experiments were taken from figure 2 of (1).]

We note that Matei et al. did show a comparison with uninitialized predictions from their model, but the comparison was against the average correlation over the entire record (2004 to 2008) rather than for each year individually as was done by Matei et al. for the “initialized” forecasts and in our Fig. 1 for both the initialized and climatological forecasts. In other words, Matei et al. presented an “apples-to-oranges” comparison of initialized and uninitialized predictions, whereas Fig. 1 puts them on an even footing.

The skill of the AMOC forecasts in (1) is no better than that from a repeating annual cycle (Fig. 1), and the skill assessment in (1) treats each year independently. That is, Matei et al. did not present any evidence of year-to-year or multiyear predictive skill in AMOC transport or its stability. Therefore, their claim that “we confidently predict a stable AMOC at least until the end of 2014” is not justified and is further contradicted by the assertion that “we cannot at present distinguish between predictability of climatological and anomalous seasonality.”

We are keen to stress that the absence of meaningful skill in AMOC predictions in Matei et al. should not be taken as evidence of an inherent lack of AMOC predictability or that predictive skill cannot be achieved in the future. Rather, our analysis highlights the difficulty in assessing skill from a short data set. Because of its scientific merit, its likely societal importance (57), and the growing model-based evidence that there may be predictability of AMOC variations (810), we believe that it is important to build our capability to understand and predict AMOC variations and associated climatic impacts. Essential to building this capability are sustained climate observations, particularly of the AMOC. Observations from the RAPID-MOCHA array (2, 3) have already identified key dynamical processes for the annual cycle of the AMOC (including some mechanisms suggested by Matei et al., which appear to successfully capture aspects of the annual cycle of the AMOC), and observations from the recent anomalous years will provide crucial information on processes controlling AMOC variability.

Development of our understanding of AMOC, its potential climate impact, and our future ability to predict it depends on sustained observations, the assessment and enhancement of GCMs, and improved methodologies for initializing GCMs for AMOC predictions. Finally and crucially, to ensure confidence in GCM predictions, a rigorous assessment of skill against reliable and meaningful null hypotheses is essential.

References and Notes

  1. We note that the P = 0.1 significance level indicated in figures 2 and 3 in (1) included an error that we correct in our version [the single-sided Student’s t test applied in (1) erroneously used the sample size, rather than the sample size minus two (11), as the degrees of freedom in the test]. The significance threshold changes from 0.55 to 0.69. A similar number (0.68) is found using the full distribution of the correlation coefficient (12) instead of its Student’s t approximation. Similarly, the significance level in figure 3 in (1) should be 0.8.
  2. Acknowledgments: We are grateful to Matei et al. for cordial discussions and exchanges in the process of preparing this Comment. We thank G. Danabasoglu, S. Griffies, W. Hazeleger, T. Knutson, J. Lanzante, G.-J. van Oldenborgh, and G. Villarini for useful comments. J.M. and E.G. were supported by the Gestion des Impacts du Changement Climatique Programme (GICC) under the Evaluation de la Prévisibilité Interannuelle à Décennale à partir des Observations et des Modèles (EIPDOM) project funded by MEDDTL (French Minister of Ecology and Sustained Development). A.K. was supported by the National Oceanographic and Atmospheric Administration under the Climate Variability and Predictability Program (NA09OAR4310163) and through a cooperative agreement with GFDL (NA06OAR4310119). J.R. received financial support from the NERC VALOR project.
View Abstract


Navigate This Article