JIFfy Pop

See allHide authors and affiliations

Science  05 Aug 2016:
Vol. 353, Issue 6299, pp. 523
DOI: 10.1126/science.aah6493

Tools for judging the impact of publications are critical for scientific leaders, librarians, and individual investigators. An important, but certainly imperfect, measure of impact is the number of times that a paper is cited over time. Citation numbers for individual papers can be aggregated to develop indices that are used to evaluate the content of journals or productivity over the course of a career. Although these indices are convenient, it is important to examine them critically to avoid their misuse.


“…how should one's publications be judged?”

A case in point is the journal impact factor (JIF), defined as the average number of times that articles published in a journal over the past 2 years are cited in a given year. JIF was developed as a metric to evaluate journals for, as an example, library resource allocation. As described recently, JIF values are based on broad and skewed distributions of citations.* I have extended these observations in my new Science blog, Sciencehound (http://blogs.sciencemag.org/sciencehound/). In my first post, I develop a mathematical function that captures the key features of the citation distributions. Subsequent posts will examine how these distributions can be used to address statistical questions regarding papers from journals with different JIF values. This analytical framework enables one to quantitatively assess the utility and, most importantly, the limitations of the use of JIFs.

As an example, take Science, with an impact factor for 2015 of 34.7. What is the probability that a randomly chosen paper from Science will have more citations than a paper randomly chosen from another journal with a JIF of 30? Modeling (to be discussed in Sciencehound) suggests a probability of 53%, barely different from chance. What about journals with JIFs of 20, 10, or 5? These probabilities are estimated to be 62%, 80%, and 91%, respectively. Even with a spread of 30 in the average number of citations per paper, the probability for concluding that a paper from one journal will have more citations than a paper from another does not reach the usual criterion of a P value below 0.05 used to judge statistical significance.

Despite this lack of discriminating power, JIFs are sometimes (ab)used to judge individual papers or scientists in some institutions around the world. The presumption is that comparison of the JIF of the journal in which a given paper appears with that from other journals or against some other standard provides substantial insight about the impact of that paper. Given the analysis of the citation distributions, this presumption is clearly invalid as a matter of mathematical fact. Thus, JIFs should not be a component of key decision processes such as faculty recruitment or promotion. This concern is independent of other criticisms regarding the robustness of JIFs, such as those noted in the San Francisco Declaration on Research Assessment, of which the American Association for the Advancement of Science (the publisher of Science) is a signatory.

If using JIFs to assess faculty is excluded, how should one's publications be judged? If a numerical metric is desired, the number of citations for a paper can be useful. This is a more direct measure of impact, particularly if factors such as the time since publication and comparative citation data from other papers in the same field are taken into consideration. More subjective measures can also be very important. Opinions about the impact of particular papers or a body of work rendered by qualified scientists in the same or similar fields have traditionally played an important role, and this should continue, ideally with appropriate consideration of potential sources of conscious or unconscious bias from these referees. Who knows? Perhaps those charged with making these important decisions should read the papers themselves, assuming that their area of expertise is close enough to the field under consideration.

Navigate This Article