Technical Comments

Response to Comment on "International Conservation Policy Delivers Benefits for Birds in Europe"

See allHide authors and affiliations

Science  22 Feb 2008:
Vol. 319, Issue 5866, pp. 1042
DOI: 10.1126/science.1151939

Abstract

Rodríguez-Muñoz et al. raise concerns about our study on the effectiveness of the European Union's Birds Directive, based on the provenance of the trend data used and on the predictions that were tested. Here, we show that our results are robust to different assumptions of uncertainty surrounding trend estimates and that criticisms of the methods stem largely from misunderstandings of the original paper.

We assessed the impact of the Birds Directive, an international bird conservation policy that covers all member states of the European Union (1), through a comparative analysis of population trends. Rodríguez-Muñoz et al.(2) rightly point out that the estimates of population trends varied in how much they were based on quantitative data, as already described in the original sources (3, 4). We were acutely aware that systematic trends in data quality might influence interpretation of results. The results published in (1) were taken from a wider set of analyses undertaken to assess the robustness of our conclusions to different assumptions of data reliability, and we regret not making this clear in the original paper. We addressed this by weighting trend estimates according to their sources or by removing trend estimates not based on quantitative data (5, 6). Using statistical methods described in (7), we initially analyzed both weighted and unweighted trends, obtaining in both cases results that would lead to conclusions identical to those reached using log-odds models. In the final analyses, we deliberately treated the data as ordinal, using the cumulative log-odds model. In this framework, any information on differences in trends is contained solely in the exceeding of thresholds of trend bands. We believe that this should reduce problems that could arise from differences in the degrees of certainty surrounding trend estimates. However, we also assessed assumptions of data quality by removing all trends not based on quantitative data. The results were virtually identical to those given by analysis of the full data set (Fig. 1), the only difference being strengthened support for our third expectation (that trends of Annex I and non–Annex I species would be more positive within the EU15 that outside it) (Fig. 1E). However, in (1) we presented only the results of the analyses of the complete data set, because this produced the most interpretable results and because excluding large amounts of data can itself introduce bias. We see no reason why contributors should have assigned a score of zero to unknown trends rather than use the “unknown trend” code that was available.

Fig. 1.

Recalculation of figure 1 in (1), showing odds ratios estimated from only those trends based on quantitative data. Interpretation is as for the original figure. The results are almost identical.

While the concerns of Rodríguez-Muñoz et al. relating to the data are understandable, although we believe unfounded, their criticisms of our methods appear to contain a number of misunderstandings. They suggest that our first expectation is invalid because of a possible, though unsubstantiated, association between population size and trend. However, their interpretation of this expectation, “after 1990, population trends for Annex I species should be less negative than those for typically more abundant (non–Annex I) species,” differs substantively from our stated expectation. The test of our expectation, although not their interpretation of it, lies in a comparison of changes in trends between the two time periods and two blocks of countries. It is unclear how any relationship between population size and trend could explain the observed reversal in the relative fortunes of Annex I species within, but not outside, the EU15 (Fig. 1A).

Rodríguez-Muñoz et al. further criticize tests of our second and third expectations as being both pseudoreplicated and unreplicated. We were not attempting to draw inferences about a wider set of countries or species using those sampled. Instead, our conclusions were based on an almost complete set of trends of all bird species in all European countries. Thus, the usual problem of pseudoreplication, a false confidence in the extrapolation of the results to a wider population, does not apply. Furthermore, our analyses accounted for non-independence in trends within and between countries, resulting in the loss of many degrees of freedom, and we would not expect any residual non-independence to influence the direction of any differences. Indeed, factors influencing the dynamics of bird populations at spatial scales larger than those of individual countries are expected to reduce any differences between groups of species or countries. Despite the testing of expectations that compared trends both longitudinally and horizontally, this is clearly an observational study rather than a designed experiment with randomly allocated treatments, so the usual caveats in interpretation must apply. In the sense that there is only one EU with one policy history, the study is inevitably unreplicated, and it is hard to envisage how the test of any international agreement could be otherwise.

The comment that “the obvious prediction that Annex I species in the EU15 should be doing significantly better than those outside the EU after 1990, but not before it, is not supported...” (2) reflects misunderstandings of both our methods and our results. This “obvious prediction” is in fact our third expectation, and Fig. 1C clearly shows that there is indeed a significant difference after 1990 (denoted by asterisks above the red bar), but not before. A statistical comparison of the red and blue bars is meaningless here. Misunderstanding of these odds ratios may underlie several of the other criticisms of Rodríguez-Muñoz et al.

In their criticism of our fifth expectation, Rodríguez-Muñoz et al. did not point out that of all the EU15 countries, Denmark was the only one that by 2000 had designated substantial areas of marine SPAs (perhaps confusing these with coastal SPAs when referring to Netherlands and Belgium). As these are not managed or protected in any way similar to terrestrial or coastal SPAs, we used just the nonmarine SPA area of each country. To do otherwise would have made the SPA cover of Denmark unrepresentative. The relationship shown in figure 2 in (1) is certainly weakened by selectively removing one of the two points in the upper right quadrant, as reflected in the marginally significant nature of the relationship. Our reason for showing this as a graph was to make clear the nature of the relationship.

Taken together, the results of tests of a number of expectations, which compared population trends within and across different groups of countries, species, and periods, and the finding that they were robust to assumptions of the nature of the data used, provide strong evidence of an impact of the EU Birds Directive. We are grateful to Rodríguez-Muñoz et al. for raising some methodological issues that we did not sufficiently clarify in the original paper and regret any confusion caused. We encourage further discussion among conservation biologists on what prestated goals should be formulated to evaluate international policy interventions and what sort of monitoring should be implemented to allow evidence-based assessment of such policies.

References and Notes

View Abstract

Navigate This Article