Policy ForumReproducibility

Enhancing reproducibility for computational methods

See allHide authors and affiliations

Science  09 Dec 2016:
Vol. 354, Issue 6317, pp. 1240-1241
DOI: 10.1126/science.aah6168

eLetters is an online forum for ongoing peer review. Submission of eLetters are open to all. eLetters are not edited, proofread, or indexed.  Please read our Terms of Service before submitting your own eLetter.

Compose eLetter

Plain text

  • Plain text
    No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Statement of Competing Interests

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

  • RE: Open Software - The need to avoid unanticipated consequences
    • Jonathan D. Linton, Chair in Operations and Technology Management, School of Management, University of Sheffield
    • Other Contributors:
      • Edward J. Daw, Reader in Gravitational Waves and Dark Matter Physics, Department of Physics and Astronomy, University of Sheffield

    Horrific depiction of future societies in movies and literature are often the result of unanticipated consequences of something(s) that seemed a good idea at the time. While we - and most other scientists - are committed to the concept of open research, it is critical that appropriate policies and regulations are in place beforehand. The first step for this is to identify the likely consequences of implementing an Open Software requirement. Three major concerns with Open Research Software policy are:
    (1) Time required to support/defend software – with increasing level of software complexity there will be a greater number of inquiries and time required to explain the software and its operation. Furthermore, misuse of the software will lead to suggestions that the software and associated research is flawed. This will require additional time and effort for the research team to defend their work.
    (2) Transfer of value associated to the software – in the United States and many other countries, any economic value derived from the researcher is the property of the research team and/or institution. An open software regime is a transfer of value from researchers and universities to other parties. This does not only economically disadvantage universities, but reduces their ability to demonstrate the contribution of research to local and national economies as there will be no technology transfer links between organizations profiting from the technology and the research te...

    Show More
    Competing Interests: None declared.
  • RE: Beyond the Reproducibility just like in Reconfigurable Manufacturing System based Mass Customization
    • Cedric Fan, Professor, MIT Information Quality Program- Data Quality & Info Security Lab, Nanjing Tech University, Nanjing, 210009, China.
    • Other Contributors:
      • Amanda Cao, Visiting Fellow, Wolfson College, the University of Cambridge, Cambridge, CB3 9BB, UK.

    In their Insight “Enhancing reproducibility for computational methods” (Science 9 December 2016: Vol. 354 no. 6317 pp. 1240), Victoria et al. (1) presented a novel set of Reproducibility Enhancement Principles (REP) for computation, which had more general proposals from the Transparency and Openness Promotion (TOP) guidelines.  

    REP tries to reduce the amount of time and resource investment necessary to facilitate reproducibility and support increasingly ambitious computational research. However, we should go beyond the Reproducibility just like in Mass Customization Manufacturing.

    From the bottom level to the top level, the ability for computation can run from Reproducibility, Reusability, Reorganizability to Reconfigurability. Beside this, the ability of Openness should also be considered including Expandability and Integrability.

    1 Victoria Stodden et al. , Science 354, 6317 (2016).

    Competing Interests: None declared.
  • RE: Another barrier to reproducibility
    • Mark W. Maciejewsk, Professor, UConn Health
    • Other Contributors:
      • Adam D. Schuyler, Professor, UConn Health
      • Michael R. Gryk, Professor, UConn Health
      • Ion I. Moraru, Professor, UConn Health
      • Pedro R. Romero, Staff Scientist, University of Wisconsin
      • Eldon R. Ulrich, Director Emeritus, BMRB, University of Wisconsin
      • Hamid R. Eghbalnia, Staff Scientist, University of Wisconsin
      • Miron Livny, Professor, University of Wisconsin
      • Frank Delaglio, Staff Scientist, University of Maryland and NIST
      • Jeffrey C. Hoch, Professor, UConn Health

    Stodden et al. (1) bring welcome attention to the problem of reproducibility of computational analyses, and recommend guidelines for addressing the barriers to reproducibility. However, one important barrier to long-term reproducibility of computational methods was not addressed. This is the problem of software persistence, in the face of rapid obsolescence of operating systems, compilers, interpreters, run-time libraries, or other software components required to execute published code. It is not sufficient to publish data, codes, and workflow meta-data without also providing the complete computational environment necessary for executing the code. This problem is particularly acute in fields where most software emanates from academic laboratories that cannot provide long-term support. Given the rapid evolution of computing systems, in a relatively short time it can become impossible to execute otherwise useful codes on extant computing platforms.
    In the field of NMR spectroscopy, NIH recently supported the establishment of a National Center for Biomolecular NMR Data Processing and Analysis (http://NMRbox.org, grant P41GM111135) to address this problem. NMRbox fosters software persistence by provisioning a virtual machine (VM) with available NMR software. The VM is regularly updated as the target operating system and the embedded NMR software evolve. Archived instances of the NMRbox VM maintained on a secure storage system are availabl...

    Show More
    Competing Interests: None declared.

Stay Connected to Science