News Focus

The Web's Faceless Judges

Science  09 Aug 2013:
Vol. 341, Issue 6146, pp. 606-608
DOI: 10.1126/science.341.6146.606

PubPeer is the latest forum for free-ranging discussion of published papers. It can only succeed, say its anonymous founders, if participants are able to keep their identities hidden.

ILLUSTRATION: DAVID PLUNKERT

What does it take to run a website where scientists can chat freely about published papers?

Anonymous e-mail addresses. Temporary phone numbers. Undisclosed locales. Jitters that one day, your cover will be blown, your career destroyed, and your family's finances depleted. It sounds like a John le Carré novel. But no, the protagonists here are a handful of biologists who last fall unveiled PubPeer, which bills itself as "an online community that uses the publication of scientific results as an opening for fruitful discussion." The goal is something of a free-for-all journal club, welcoming comments from readers and authors across disciplines.

"When I saw that it was not signed by anybody, I felt uncomfortable—it was an instinctive reaction".

—Shaul Hestrin, neuroscientist at Stanford University

PubPeer is one of several recent ventures to encourage scrutiny of published work, seeking to fill what many consider a gap in scientific publishing. "I myself have kind of longed for a place where I can see discussion of work after publication," says Ferric Fang, a microbiologist at the University of Washington, Seattle, and editor-in-chief of Infection and Immunity. Journals are one obvious place to leave comments about a paper, says Fang, who has written extensively about scientific publishing and misconduct (Science, 25 January, p. 386). But he says that most "haven't been very good" at nurturing such discussion. Some don't allow comments at all, and others require commenters to be named or remove those that may imply wrongdoing.

When questions about published research bleed into misconduct accusations, journals and institutions have their protocols, but many researchers grouse that the process can take years and its outcome is often unsatisfying. "Universities charged with investigation [have a] huge conflict of interest," says Jennifer Nyborg, a biochemist at Colorado State University, Fort Collins, who's been frustrated by her efforts to report potential misconduct through the proper channels.

Given these shortcomings, many agree there's a place for sites that engage in postpublication peer review. They can clarify experiments and catch errors, something several, including PubPeer, have done. They can challenge how studies are interpreted and suggest avenues for follow-up work.

But many who participate in these discussions sit at a tense nexus: They long for more unfettered conversation about science, yet insist on doing so anonymously, fearful that their words will come back to haunt them. One of PubPeer's founders, who describes himself as a tenured professor, says that even a senior scientist "very rarely, myself included, wants to take the risk" of criticizing fellow scientists under their own names. The professor and his shadowy brethren—another founder tells Science that he is finishing up his Ph.D. somewhere in the United States—have gone to great lengths to protect their identities. "I don't want it to impact my scientific life or my personal life," says the professor of his site, adding that the phone number from which he was calling "probably won't work after a few days."

While anonymity can spur discussion, it does not always elevate it.

When PubPeer launched in October 2012, the founders' goal was genteel dialogue. "I enjoyed this paper greatly," an anonymous commenter wrote early this year, about a study in Science on empathy in rats. The commenter politely queried about comparisons between littermates and nonlittermates and sought more information about how the animals behaved.

Posts like this one, though, were interspersed with those of a different nature. "The paper was VERY effective for getting his lab a lot of publicity (and money?)," one commenter wrote about an article in Scientific Reports, a Nature journal. "Was it just sensationalism or did it tell us something new about the brain?"

In May, PubPeer underwent a tectonic shift after a tip exposed errors in a high-profile paper. The anonymous post flagged apparent image duplications in a manuscript describing how human embryonic stem cells could be produced by cloning, published in Cell by a group based in Oregon (Science, 31 May, p. 1026). Cell subsequently printed an erratum. Suddenly, PubPeer was in the news, and whistleblowers began flooding it with tips suggesting problematic images in dozens of papers. It now receives between 10 and 50 comments a day. "They're getting a little bit out of control, in my opinion," the professor founder says.

"The volume of material they're dealing with, I think, is far, far greater than anyone anticipated," agrees Paul Brookes, a biologist at the University of Rochester in New York. Brookes knows well what it takes, and what it's like, to play scientific watchdog. Last year he launched www.science-fraud.org, which unlike PubPeer sought from the start to expose problematic images in published papers. Brookes, too, was anonymous—until January, when a still-unidentified individual outed him (Science, 11 January, p. 132). Amid threats of lawsuits, science-fraud.org imploded. Brookes quickly pulled accusations of wrongdoing from the site. Although many criticized him for questioning the integrity of authors, the site had an impact: More than 40 of the papers that appeared on science-fraud.org have since been corrected or retracted, Brookes says.

So far PubPeer's record is slimmer, but the site appears to be gaining traction. In July, it posted 89 comments, up from just 26 in June. In May, when the Oregon stem cell case broke and science-fraud.org posted an endorsement of PubPeer, comments on the site topped 360. Some challenge the integrity of images; others question whether a paper's conclusions are backed by its data. Scientists who are first or last authors on a paper can register as anonymous "peers" and leave comments, which are posted automatically but can be removed if deemed inappropriate. Anyone can send unregistered submissions to PubPeer's e-mail address, and the founders often vet their claims against the paper in question. (In June and July, more than 500 comments were submitted but not posted due to problematic tone or content.) There's also the option of commenting under one's name, but few take that route.

"If it's a verifiable fact, … it just shouldn't matter" if the person is named, says Ivan Oransky, a journalist who 3 years ago started the popular blog Retraction Watch with fellow reporter Adam Marcus. But he says that in some intangible way, "it does. People like to know where people are coming from." Many of the tips about retracted papers that he and Marcus investigate arrive under pseudonyms, from anonymous e-mail addresses. Even though Oransky supports anonymity, he admits that he can't help but be curious himself, especially about those who attack Retraction Watch and accuse it of having "an agenda." "I would very much like to know who all these people are," he says.

"If it's a verifiable fact, … it just shouldn't matter" if the person is named. But "people like to know where people are coming from".

––Ivan Oransky, co-founder of Retraction Watch

A big reason for staying hidden, many scientists suggest, is that despite all the talk of honest discussion in their community, there's little reward for engaging in it. "If the system was much more open and much more tolerant of dissent then this would not be needed," says Raphael Levy, who studies cell imaging at the University of Liverpool in the United Kingdom. He has left both anonymous and named comments on PubPeer.

Another occasional commenter, neuroscientist Boris Barbour at the Institute of Biology at École Normale Supérieure in Paris, concedes that his writing changes when it's backed by his name. "It's very easy to say, 'That paper's crap and how did it ever get accepted,'" Barbour says. "To be sure you have a water-tight case isn't trivial." Although he tries to be equally rigorous whether posting anonymously or not, he says the pressure to "make extra certain" the text is accurate is heightened when he goes public. Fang, who has left comments on Retraction Watch, agrees. "When I have to sign my name to it, it makes me just a little bit more thoughtful before I hit 'send,'" he says.

When a paper is flagged on PubPeer, the site immediately notifies the corresponding author. Peter Klein, a physician and developmental biologist at the University of Pennsylvania, received an automated e-mail from PubPeer in late May. A poster listed 18 questions and comments about his 2012 report in Nature Medicine, including what this person described as "contradictions between text and figures" and "incomplete or inconsistent description of methods and figures."

Klein had never heard of the site. "Because there were so many small details, my first response was, do we have to respond? We've already gone through peer review," he says. "But then as I went through it … it occurred to me that this person probably found the omissions frustrating. … I felt like, we can answer them, we can help them out." He and his postdoc crafted a nearly 900-word rejoinder.

Few authors were as relaxed about anonymity as Klein. "When I saw that it was not signed by anybody, I felt uncomfortable—it was an instinctive reaction," says Shaul Hestrin, a neuroscientist at Stanford University in Palo Alto, California, whose paper in The Journal of Neuroscience was questioned. Hestrin sensed an imbalance, because only one person was named. (That would be him.) "Either you want to stand by what you're saying, or you don't say it." Hestrin e-mailed the site to say he'd be happy to respond if he knew who was asking the questions, but was told that wasn't an option. "I just disengaged," he says, and didn't reply, although he acknowledges the points raised—essentially, whether the data had been overinterpreted—weren't unreasonable.

Ironically, some of those who decry anonymity most vigorously insisted that their names not appear in this story. "Anonymity's a great cover for people who want to take other people down," says one author whose work was cited on PubPeer. Another scientist argued that PubPeer "becomes basically a gossip site" lacking "credibility or accountability." ("I've never done anything anonymously," this person emphasized, while stipulating that their name not be publicized.)

Brookes, who is facing several threatened lawsuits, has thought a great deal about anonymity. His views are still evolving. "Previously I think I was very much like the owners of PubPeer—very scared, very wary," he says. "And I'm sort of coming around to the idea that doing this stuff using your real name is the way to go."

Brookes cites a couple of reasons for the swing of his internal compass. For one, he was frustrated when he tried to contact a journal anonymously about potential image manipulations in five papers by a single author. After dozens of e-mails generated not a single reply, he became convinced that the reason was that no one knew who he was.

Brookes also notes an important change in his own scientific fortunes: Last year, while running science-fraud.org, he was applying for a grant from the National Institutes of Health. In February, he learned that his application would be approved. "It's really down to what career state you're at and what you're comfortable with," he says. Brookes is contemplating a new site for postpublication peer review. He's still considering the place anonymity might have on that site.

PubPeer's founders watched science-fraud.org's collapse warily. "We learned from that, we're not accusing anybody of fraud," says the professor founder. The site's administrators, whom he says number about eight spread across different institutions, have consulted with some family friends who are attorneys "but we are mostly winging it."

At the same time, PubPeer is doing what it can to raise its profile. It's planning to offer its users browser plug-ins that can link PubPeer comments to PubMed, which thousands of researchers rely on to sift through biomedical papers. Its administrators are also developing citation software plug-ins that would alert users to comments on papers when those papers are referenced in manuscripts or grants. "Our dream," he says, "is that this becomes an important side piece to the scientific literature."

The professor has already had occasion to be grateful for his double life. A visitor grading his lab for funding had a paper flagged on the site; a department colleague had research discussed there. "If he knew I was involved, it would change our interactions," he says, referring to the colleague. Still, he's resigned to a day when he's unmasked. "I think it's going to be very hard," he says, "to stay anonymous forever."

Related Content

Subjects

Navigate This Article