Replication challenges

See allHide authors and affiliations

Science  06 Sep 2019:
Vol. 365, Issue 6457, pp. 957
DOI: 10.1126/science.aaz2701

Embedded Image

The ability to test validity by replicating experiments and comparing results is a cornerstone of science. The events that followed publication of “Sustained virologic control in SIV+ macaques after antiretroviral and α4β7 antibody therapy” by Byrareddy et al. in Science in 2016 (hereafter Byrareddy) provide insights into some challenges associated with replication. Byrareddy reported striking efficacy of an approach for controlling simian immunodeficiency virus (SIV) in rhesus macaques in a sustained manner, a result of potential importance for treatment of human HIV disease. Three papers in this issue report attempts to replicate these results. The matters associated with Byrareddy and these studies provide a concrete framework for discussing our policies related to replication studies.

Embedded Image

Underscoring the importance of replication, the leadership at the U.S. National Institute of Allergy and Infectious Diseases, some of whom were involved in both the conduct and the funding of the Byrareddy research, decided to move forward with replication studies. Early in these efforts, the SIV strain used in Byrareddy, which was described as “SIVmac239,” was sequenced, demonstrating that the strain used had a stop codon in the SIV Nef gene. Subsequent communications with Science revealed that this was a conscious choice of the coauthor who supplied the virus, with the rationale that the virus with this variation is a better model for chronic human HIV infection. This fundamental aspect of the experimental design was not effectively communicated across the author team nor to the readers of Science. This lack of clarity is an unacceptable communication failure that affected interpretation of the results both before and after publication; Science has now corrected the paper.

Two of three attempted replications used the same SIVmac239-nef-stop strain, whereas the third used a different strain (SIVmac251) that lacked stop codons in any SIV gene. However, the results of all three studies were similar to each other and quite different from those reported in Byrareddy, with no durable control of SIV upon discontinuation of antiretroviral drug therapy. We have added an Editorial Expression of Concern so that all readers are aware of the failure of these attempted replications.

These experiences reveal some of the challenges related to replication. The Byrareddy authors have not provided any experimentally supported explanation that accounts for the differences between their observations and those from the replication studies but, nonetheless, stand by their results. These complicated biological studies involve many variables in addition to the viral strain, including the immunological experiences of the experimental animals, the methods used for viral exposure, and other factors. Even matching these factors as closely as possible, hidden variables might exist that cause differences. However, experiments that conceptually match the original study but differ in other ways can probe the robustness of the results, and such robustness can affect the likelihood that results might be translated into practice. Results of a clinical trial in 20 HIV-infected individuals treated with α4β7 antibody therapy, conducted in parallel with the replication studies and just published in Science Translational Medicine, show no durable HIV response.

The facilitation and communication of replication studies for important results is a challenge for the scientific and publishing communities. The policy of the Science family of journals is to encourage the submission of replication studies that provide new insights into previously published results. These submissions are held to the same standards as other content, with key factors including the likelihood that these papers will be influential in their fields or across fields. The novelty of the question is not a consideration, but continued high interest in the question is. Priority may be lower for cases in which the original paper would be unlikely to meet current community standards. More examples of the implementation of these policies to specific cases will help provide clarity to the scientific community.

View Abstract

Navigate This Article