Letters

Assessing Clinical Trial Results

Science  21 Apr 2006:
Vol. 312, Issue 5772, pp. 365b-366b
DOI: 10.1126/science.312.5772.365b

In her policy forum “Clinical trials results databases: unanswered questions” (13 Jan., p. 180), C. B. Fisher warns of several undesirable effects that might result from open access to raw data from clinical trials. Referring to the editorial policy of a new journal, Fisher suggests that “lack of emphasis on the direction of results or size … risks diluting scientific standards for peer review.”

In fact, neither the size of a trial nor the direction of its results in itself determines the trial's scientific validity. Certainly, it is vital that trials based on small numbers of participants and trials delivering negative results should not be overinterpreted. But, if properly conducted and controlled, small trials and trials with negative results can both make important contributions to medical knowledge.

For example, the sample size of a certain trial may be large enough to allow general conclusions to be drawn but may not have sufficient power to distinguish effects on a particular age group. By taking the raw data of several such small trials together, though, it may be possible to safely extend the conclusions beyond those of the original trials. The inclusion of results from trials with both positive and negative results is vital to such meta-analyses to ensure they are not statistically skewed.

Fisher also wonders whether “the availability of large bodies of data from studies that may or may not have scientific merit will improve or distract from the peer-review process.” Recent events in South Korea and elsewhere strongly suggest that making additional raw data available to peer reviewers whenever possible would be desirable. While not eliminating the possibility of fraud, it would at least make it less straightforward and so, arguably, less tempting.

CREDIT: PHOTODISC GREEN/GETTY IMAGES

Finally, Fisher worries that, by making available early data from clinical trials, drug companies may fall afoul of regulations relating to “forward-looking statements.” However, the data from trials are not forward-looking statements, they are reports of concrete past events. It is difficult to see how making these records available in a transparent fashion could be seen as misleading. Of course, some investors may overinterpret promising early results, just as they may overinterpret a profitable first quarter, but if companies clearly warn against overinterpretation, they should not be held accountable for it.

Any science, including published peer-reviewed research, may be abused by misinterpretation. This should not be used as a justification for hiding important data behind closed doors.

In her policy forum “Clinical trials results databases: unanswered questions” (13 Jan., p. 180), C. B. Fisher equates negative trial results with poorly conducted trials. There is no evidence that this is the case. Rather, the existence of a bias toward the publication of positive outcomes is not only well known (1), but documented examples underscore its effects: Inclusion of unpublished data can sometimes shift assessment of a treatment's value from one of overall benefit toward overall harm (2).

The new Public Library of Science journal PLoS Clinical Trials will address this problem by publishing reports irrespective of a trial's outcome. Fisher suggests that this approach “risks diluting scientific standards for peer review.” This statement is not correct; in fact, the journal will ensure the highest standard of reporting by adhering to CONSORT (3), by requiring that all trials be registered, and by obtaining a copy of the original trial protocol, which will enable reviewers to compare the final report against what was planned. All trials will be scrutinized by at least one subject-specific reviewer as well as a statistical expert, together with academic editors and in-house editorial staff. In this way, the journal will uphold and advance scientific standards for publication of trial results.

References and Notes

  1. 1.
  2. 2.
  3. 3.

The interesting policy forum by C. B. Fisher “Clinical trials results databases: unanswered questions” (13 Jan., p. 180) contains some misconceptions that warrant comment.

First, the overarching rationale for full disclosure of trial information has been to fulfill ethical obligations to trial participants, who are subjected to potential risks in exchange for the creation of public knowledge. Although potentially important, other competing concerns raised by Fisher must be subsidiary to this fundamental ethical responsibility.

Second, Fisher confounds trial results with trial methods when discussing the validity of peer-reviewed, open-access publications. An appropriately designed study is scientifically valid regardless of its findings or size alone. Small studies may provide less precise yet valid results; precision can be increased by pooling data across studies through meta-analysis. Thus, the publication of all properly conducted trials regardless of the nature or magnitude of their results will help to address the biases associated with selective reporting of research.

Third, an unjustified assumption underlying much of the paper is that peer-reviewed results are inherently better—an unresolved issue that is neither new nor specific to results databases (1). On the basis of empiric evidence, the scientific value of peer review remains unclear (2, 3). Nevertheless, references to trial results should thus continue to indicate their data source to distinguish peer-reviewed and non-peer-reviewed results.

What is clear, however, is that critical appraisal of trial methodology remains vital for placing a set of results into its appropriate context. Sufficient methodological information must therefore accompany all trial results, whether in a results database or journal.

Full results disclosure will improve the status quo of subjective, selective reporting and data suppression. Being fully transparent with numerical data is the only solution free of subjective judgment. As the global, neutral body with legitimacy and international support, the World Health Organization International Clinical Trial Registry Platform (www.who.int/ictrp) will continue its leadership role in consulting all constituencies to define global norms and standards for trial registers and results databases (4).

  • †Member of the Editorial Board of PloS Clinical Trials

  • ‡Member of the Advisory Board of PloS Clinical Trials

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.

Response

I am pleased that my policy forum has prompted discussion on the value of clinical trials databases and ways to ensure that, whether created by industry, government, or international organizations, they benefit all who have a stake in the development of safe and effective health products.

Cockerill and Norton and Chan and colleagues write that small-sample, well-designed studies may have sufficient statistical power to produce scientifically valid results—and that failure to disprove a null hypothesis may sometimes challenge a well-established body of evidence. However, as both Letters admit, there are risks in such data being overinterpreted, especially by patients and physicians with little training in scientific methodology. My Policy Forum is meant to challenge those who create public databases to work to ensure that the equal availability of poorly and well-designed or implemented studies does not create confusion or reduce public trust in medical science. Similarly, although Chan et al. criticize my assumption that peer-reviewed results are inherently better, they nonetheless suggest that it will be important to distinguish peer-reviewed from non-peer-reviewed results in clinical trials databases.

Cockerill and Norton and Chan et al. emphasize the value of clinical trials for specific types of stakeholders. I agree that medical science benefits when other scientists can evaluate raw data from published studies and that one advantage of transparency is that trial participants can learn how their efforts contributed to public knowledge. However, the range of individuals who will be affected by the quality of clinical trials databases goes beyond these two segments and includes the general public, patients, practitioners, health management organizations and other third party payors, and federal and private sponsors of research. For example, although investigators and sponsors may recognize that reporting of early data is not meant to be “forward-looking statements,” various proposals for databases that include public summaries or implications of early results may inadvertently push sponsors in that direction or create a loophole in current regulations meant to protect the public.

In response to Veitch's Letter, it is good to see more detail on the steps that PLoS Clinical Trials will take to increase transparency of research reporting. However, the additional information she provides does not change the value of the question posed in my Policy Forum. What will be the effect on established scientific standards of peer review and public confidence in these standards when investigators and the public are inundated with trial results that fail to confirm or disconfirm hypotheses, yield small effect sizes, or use sample sizes insufficiently powered to generate scientifically valid conclusions?

Timely and transparent reporting of clinical trials results is essential to public health and public confidence. To fulfill this mission, those who are lobbying for or creating clinical trials databases must ensure that such sites are of equal benefit to the multiple stakeholders in the development of health products worldwide.

Related Content

Navigate This Article