Experimental Turk

(Updated March 4, 2018) This page is in constant update [EDIT: not really] and contains papers, tutorials, and links to other resources for clever experimentation on MTurk.

Tutorials

Excluding MTurk workers who participated in your previous studies: An Excel solution
Gabriele Paolacci (Erasmus University Rotterdam)
Johannes Boegershausen (University of British Columbia)

Selectively Recruiting Participants from Amazon Mechanical Turk Using Qualtrics
Eyal Peer (Br-Ilan University), eyal.peer@biu.ac.il
Gabriele Paolacci (Erasmus University Rotterdam)
Jesse Chandler (Mathematica Policy Research & University of Michigan)
Pam Mueller (Princeton University)

(Suggested citation: Pe’er, Eyal, Paolacci, Gabriele, Chandler, Jesse and Mueller, Pam, Selectively Recruiting Participants from Amazon Mechanical Turk Using Qualtrics (May 2, 2012). Available at SSRN: http://ssrn.com/abstract=2100631 or http://dx.doi.org/10.2139/ssrn.2100631)

Emailing workers using Python
Pam Mueller (Princeton University), pamuelle@princeton.edu
Jesse Chandler (Princeton University)

(Suggested citation: Mueller, Pam and Chandler, Jesse, Emailing Workers Using Python (March 3, 2012). Available at SSRN: http://ssrn.com/abstract=2100601)

Review Article: 

• Paolacci, G., Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a Participant Pool. Current Directions in Psychological Science, 23(3), 184-188.

(Some) Fundamental articles on MTurk experimentation

• Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.

• Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

• Horton, J.J., Rand, D.G., & Zeckhauser, R.J. (2011). The Online Laboratory: Conducting Experiments in a Real Labor Market. Experimental Economics, 4, 399-42

• Suri, S., Watts, D. J. (2011). A Study of Cooperation and Contagion in Web-Based, Networked Public Goods Experiments, PLoS ONE 6(3).

• Amir, O., Rand, D., Gal, Y.K. (2012). Economic Games on the Internet: The Effect of $1 Stakes. PLoS ONE 7(2)

• Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com’s Mechanical Turk. Political Analysis, 20(3), 351-368.

• Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1-23.

• Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224.

• Crump, M.J.C., McDonnell, J.V., Gureckis, T.M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8(3).

• Chandler, J., Mueller, P., Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior Research Methods. 46(1), 112-130.

Online resources

• Sid Suri and Winter Mason maintain a wiki about research on AMT and other crowdsourcing platforms.
• Michael Buhrmester maintains a page about conducting experiments on AMT.
• Panos Ipeirotis’ blog is often posting about AMT.
• Dave Rand gave an introductory talk about conducting experiments on AMT.
• Adam Finkelstein’s course “Crowdsourcing your experiment” refers to many useful resources to run experiments on AMT.
• Deneme is another blog on experiments on AMT, with a different purpose. It also provides a link to Turkit, a Java/JavaScript API for running iterative tasks on AMT.