Policy ForumHealth-Care Policy

To Reform U.S. Health Care, Start with Systematic Reviews

See allHide authors and affiliations

Science  30 Jul 2010:
Vol. 329, Issue 5991, pp. 516-517
DOI: 10.1126/science.1189330

With passage of The Patient Protection and Affordable Care Act (Public Law 111–148), the United States has taken an important step toward health care for all. But can “good” health care for all be achieved? The United States has lagged behind Canada, Europe, Australia, and now China (1) in making a national commitment to creating and utilizing research evidence on “what works” as an integral part of clinical practice and health policy (2). This approach, termed “evidence-based health care,” has been defined as “the integration of best research evidence with clinical expertise and patient values” (3). Evidence-based health care is not without its critics. Some have rejected it as “cookbook medicine” (4, 5), apparently ignoring that evidence is not the only ingredient in the approach and that the alternative ignores the scientific underpinnings of medicine.

Evidence We Need, Evidence We Have

For good health care, the research evidence we need most is from studies of interventions aimed at treatment, prevention, diagnosis, and screening. Sometimes the evidence we seek, for example, from randomized trials, is scarce or difficult to identify or its findings are inconclusive. We can solve these problems, which together constitute a research “evidence gap,” but first we have to acknowledge them and work in a coordinated fashion with our international colleagues so that we get the answers we need quickly, without unnecessary duplication of effort.

We have plenty of research, published in over 22,000 biomedical journals, so why do we not have the answers we need? When evidence is unreliable, does not exist, or is insufficient to address important questions, we must generate new evidence. But some have argued that, before we initiate new research, we should be required to show whether it is needed by conducting a formal synthesis of currently available knowledge and that similar credit should be accorded to those synthesizing research as to those performing primary research (68).

“Systematic reviews” are a scientific approach being used across the sciences (1115) to synthesize existing evidence. When it appears that studies are similar enough in the questions they are addressing and that they are of sufficient quality, results of the studies may be combined quantitatively in a “meta-analysis” (12, 14). Meta-analysis is an optional quantitative synthesis within a systematic review. There are agreed-upon criteria for designing high-quality primary studies (e.g., control group, adequate sample sizes, and transparent reporting). Ought we not to insist on the same for review articles?

Synthesizing research evidence is challenging and requires training (2). In addition, even proponents of systematic reviews are concerned that poor-quality primary studies result in unreliable synthesized results and that combining results of studies that are too dissimilar (e.g., in population, interventions, and outcomes assessed) can lead to erroneous conclusions (14). At the direction of the U.S. Congress, the Institute of Medicine is currently producing standards for systematic reviews, an important step (15).

As far as I know, no NIH institute is prepared to take on a large-scale, deliberate, coordinated effort to generate evidence and to conduct systematic reviews across health fields and topics. The National Institutes of Health (NIH), with few exceptions, has focused on funding the generation of evidence, not its synthesis, which is arguably equally important because it reliably identifies areas where new evidence is needed. It is not clear that any NIH institute has required a comprehensive systematic review of evidence before a clinical trial is funded, beyond the literature review requested as a part of the grant application. Given the high cost of clinical trials, it seems fair to ask applicants for a systematic accounting of what is known before funding is approved, such as is required by the UK Medical Research Council (16) and others (17).

When there are coordinated U.S. efforts for performing systematic reviews, funding agencies have focused mainly on federal programs and have used internal agency staff or contractual investigators (18). These programs [e.g., (19, 20)], although critically important, lack capacity and were never intended to cover the full range and depth of health-related questions.

Addressing the Research Evidence Gap

One alternative approach, which might both increase capacity and reduce duplication of effort, would be for the United States to increase its investment in international activities aimed at synthesizing the available evidence. The Cochrane Collaboration (21) engages people from more than 100 countries to conduct and maintain systematic reviews of the best current evidence to address important questions relating to clinical care and health policy. Since its beginnings in 1993, Cochrane has also played an important role in developing standards for conducting and reporting systematic reviews (2124). The global, nonprofit Cochrane effort has been likened to the health services equivalent of the Human Genome Project (25).

The decision to conduct a Cochrane review may emanate from an author's interest in the topic (“voluntary”), from a prioritization exercise (“commissioned”), or a combination of the two (e.g., prioritized topics are made public and authors volunteer). Cochrane reviews are made publicly available through an electronic journal, The Cochrane Library, updated monthly and currently made freely available to more than half of the world's population, through national subscriptions or other provisions, for example, to low-income countries (26). The collaboration has long-standing, freely available standards for conducting reviews (12); is among the most productive groups in the world, in terms of number of reviews produced and updated; and is often cited (27). As of July 2010, The Cochrane Library includes 4309 completed reviews and 1935 protocols of reviews in preparation. The Cochrane Collaboration also enlists a well-trained workforce, with more than 1500 of its affiliated systematic review authors based in the United States.

In addition to investments federal funding agencies are already making in their own programs, they can and ought to invest in a coordinated way in infrastructure for Cochrane systematic reviews and to encourage cooperative work between knowledge producers to increase efficiencies. Although the Cochrane Collaboration itself does not pay for salary support for review authors, all are supported indirectly by the collaboration infrastructure, which provides centralized management, standardized guidance on methods, quality control, education, and support for consumer engagement. No commercial funding of Cochrane reviews is permitted. Furthermore, because Cochrane entities are funded through many sources and governments, any single funding source (such as the NIH) can leverage its own investment against global investment.

However, federal funders may want more control over the project than that which appears possible through an existing global, not-for-profit effort. Related to this, there can be a misperception that Cochrane is a European organization, and a concern that a non-U.S. focus could dilute U.S.-based goals (28). Although a few promising efforts have taken place [e.g., Cochrane support provided by the National Institute of Child Health and Human Development (29) and the National Eye Institute], it is not clear that appropriate grant mechanisms are in place that would allow a broad-based, coordinated investment in the existing U.S.-based Cochrane infrastructure; the newer Common Fund (30) and Blueprint (31) mechanisms that support trans-NIH initiatives may be useful models for such a broad-based initiative. The Cochrane Collaboration may have to change too, perhaps to be better able to understand and respond to funder priorities.

Most systematic reviews have been done poorly in the past (32), which indicates a need for skilled faculty and educational programs that will yield a capable and dedicated workforce. The United States can tap into Cochrane reviewers' existing, underutilized expertise. In addition, research groups doing systematic reviews must be developed and sustained so that a group's knowledge and skills are not lost with the completion of a review. Furthermore, to attract and retain creative and productive investigators, research synthesis must be recognized as a methodology and field in its own right, and systematic reviewers must be given adequate credit for their research (33).

The Way Forward

The Patient Protection and Affordable Care Act mandates creation of a new, nonprofit corporation, the Patient-Centered Outcomes Research Institute (PCORI), and a funding stream for “comparative clinical effectiveness research” (CER)—both new primary research and research synthesis—comparing the benefits and harms of interventions to prevent, diagnose, treat, and monitor health conditions. The institute holds promise to coordinate efforts to produce good science and to serve as the flagship for collaborative, productive research. Specific targets of investment would be as follows:

  • Take advantage of being late to the game and test the most promising approaches to health care identified elsewhere. The United States cannot afford a “not invented here” mindset, ignoring lessons and promising models from outside the United States and, instead, building from scratch.

  • Provide support for a coordinated approach to performing high-quality systematic reviews, both commissioned and voluntary.

  • Develop, expand, adapt, or clarify federal funding mechanisms to ensure that support is sustained so research groups do not disband as systematic reviews are completed, and so reviews are kept up-to-date.

  • Provide federal infrastructure support to external scientific organizations, such as the Cochrane Collaboration, so as to leverage international collaboration.

  • Commit substantial and sustained resources to training grants specifically aimed at educating and expanding the professional workforce required to synthesize the evidence and keep reviews up-to-date.

  • Increase federal support for research on methods for more efficient and informative trials and systematic reviews.

  • Provide support for knowledge translation research, to learn how best to translate research evidence so that it is useful to decision-makers. Although PCORI provides a budget for “dissemination” of CER findings, it is not clear that there are funds for research to identify effective ways to do this.

  • Involve health-care decision-makers (e.g., consumers, policy-makers, and doctors) throughout the research process so that the evidence generated is likely to be useful.

Synthesis of what we know from primary research, using up-to-date systematic reviews, is critical to each step in the process, from prioritizing research to implementing high-quality, cost-effective clinical care. Changes in the federal funding and academic reward systems could help attract, educate, and retain talented researchers and support high-quality research and infrastructure that will help lead to evidence-based health care.

References and Notes

  1. K.D. gratefully acknowledges comments by M. Redford, K. Robinson, and D. Tovey and contributions of Cochrane data from N. Royle and L. Jones.
View Abstract

Stay Connected to Science

Navigate This Article