Technical Comments

Comment on "International Conservation Policy Delivers Benefits for Birds in Europe"

See allHide authors and affiliations

Science  22 Feb 2008:
Vol. 319, Issue 5866, pp. 1042
DOI: 10.1126/science.1151810

Abstract

Donald et al. (Reports, 10 August 2007, p. 810) assessed the impact of the European Union's Birds Directive, a conservation policy enacted in 1979, and reported evidence for positive population changes in targeted species. We argue that their conclusions are overstatements based on unsuitable data and inappropriate analyses.

In the absence of any systematic program to determine the effectiveness of the European Union's Birds Directive, an international policy instrument introduced in 1979, Donald et al.(1) attempted to determine whether it has achieved its aims, using publicly available data. Their approach involved dividing the years between 1970 and 2000 into the 20 years before and 10 years after 1990. They then compared population trends over these periods for birds listed in Annex I of the directive (species considered to be vulnerable or rare, or to require special conservation measures) with trends for unlisted birds. We have two classes of criticism regarding their study: first, that the data are misused or unreliable, and second, that the predictions tested are not robust.

Donald et al. appear to have used all available data in two publications (2, 3), each covering one of the two set time periods. Data in these publications are described as coming from published and unpublished sources, including “relevant experts, monitoring organizations, and regional contributors” [see supporting online material in (1)]. However, the cited reports are cautious about the accuracy of their data, dividing it into three classes of reliability, including a large class described as “poor non-quantitative data” for which “caution in the use of individual population or trend figures” is advised (2). Poor quality data can lead to spurious conclusions, but this is only a real worry if there are systematic differences in data quality between groups being compared. Unfortunately, this appears to be the case in the analysis in (1). We examined data quality as designated in (2, 3) using the Birds in Europe online database (4) and found that data from the original 15 member countries of the EU (EU15) are of substantially higher quality than data from non-EU15 countries and that data for species included in Annex I are of higher quality than data for non–Annex I species (Table 1). Even more problematic is that data associated with positive population trends are of substantially higher quality than data interpreted as revealing negative trends (Table 2). These types of systematic bias are cause for concern for reasons including the likelihood that when data are scarce, ornithologists may interpret population trends as more negative than when there is less room for subjectivity.

Table 1.

Comparison of data quality between EU15 and non-EU15 countries for Annex I and non—Annex I species. Asterisks denote significant differences after Mann-Whitney U tests (*P < 0.05, **P < 0.01, ***P < 0.001).

View this table:
Table 2.

Results of Kruskal-Wallis One-Way Analysis of Variance to compare data quality between three possible trends (increasing, stable, or decreasing) for different groups (KW values). In all cases, quality was highest in groups with increasing trends (*P < 0.05, **P < 0.01, ***P < 0.001).

View this table:

In addition to these data quality issues, we have concerns about four of the five expectations described by Donald et al. as criteria for assessing the efficacy of the Birds Directive. The first prediction is that after 1990, population trends for Annex I species should be less negative than those for typically more abundant (non–Annex I) species. However, this prediction is confounded because rates of change in abundance are unlikely to be independent of population size, and we might expect declines to slow as populations become very small regardless of conservation efforts. The next two expectations are that this effect should be greater in the original EU countries, and that trends for all bird species should be greater in the EU15 than outside it. The problem with these predictions is that comparison of EU15 versus non-EU15 countries is pseudo-replicative [sensu (5)] because the EU15 countries typically have more in common with each other in relation to several factors that affect conservation than they do with countries outside the EU15 (not least because of the influence of the EU itself). Although Donald et al. attempt to address some potential confounds, they cannot get around this lack of replication, which means that little confidence can be placed in their test. It is also worth noting that the obvious prediction that Annex I species in the EU15 should be doing significantly better than those outside the EU after 1990, but not before it, is not supported; there is no significant difference [see figure 1C in (1)].

For the fifth expectation, Donald et al. hypothesized that bird population trends would be more positive in countries with a higher degree of implementation of the Birds Directive. They stated that they took data on total land area covered by Special Protection Areas (SPA), excluding marine SPAs, from the April 2000 Natura 2000 newsletter (6) and used this as a measure of delivery of conservation. They did not discuss why marine SPAs were excluded given the large number of seabirds in Annex I. We reanalyzed these data and found that if total percentage of SPA (%SPA) is used as a measure, the trend is nonsignificant (r2 = 0.125; P = 0.196). In fact, the April 2000 Natura 2000 newsletter only provides total %SPA; the source of the information on terrestrial %SPA is not provided. It appears that rather than excluding all marine SPAs, as stated, only the figure for Denmark has been replaced by the figure for the terrestrial area published in the October 2004 newsletter (7), although this is not cited. For the other 14 countries, figure 2 in (1) presents the same %SPA values as in the April 2000 news-letter, despite the fact that countries such as The Netherlands and Belgium also have large marine SPAs. Using terrestrial %SPA from 2004, the relationship, again, is nonsignificant (r2 = 0.158; P = 0.143). Overall, the relationship between population trends and %SPA only becomes significant when using total %SPA in 2000 for 14 countries and substituting the value for Denmark with that of terrestrial %SPA in 2004. This approach is not justified or even made explicit.

The issues outlined above raise serious questions about robustness of the conclusions in (1). We wholeheartedly agree with Donald et al. that reliable information about the success or failure of European conservation policies requires integrated monitoring and careful analysis. We understand the urgent need for feedback on the effects of conservation policies, and it would be easy to argue that Donald et al. are attempting to do the best they can with the available data. However, policy makers will not read caveats buried in discussions and appendices. Using inappropriate data sets and analyses, even if better tools are not available, risks undermining efforts to promote appropriate monitoring of the outcomes of conservation policies.

References and Notes

View Abstract

Navigate This Article