Psychology

Sampling online workers globally

+ See all authors and affiliations

Science  10 Mar 2017:
Vol. 355, Issue 6329, pp. 1036-1037
DOI: 10.1126/science.355.6329.1036-d

Historically, experimental psychologists have relied heavily on undergraduates for their subject pool; over the past decade, they have recruited subjects online, employing people colloquially termed MTurkers through the Amazon Mechanical Turk website. Over the years, MTurkers have become so familiar with online surveys that there are concerns about the representativeness and reliability of these participants. Peer et al. compared MTurk with two other crowdsourcing sites, CrowdFlower and Prolific Academic. The latter two sites produced participants who were less familiar with the kinds of tasks assigned (good) but also less attentive to instructions (bad), and they had more European participants. In terms of reliability and reproducibility, Prolific Academic scored higher than CrowdFlower and about the same as MTurk.

J. Exp. Soc. Psychol. 70, 153 (2017).

Navigate This Article