(Updated May 2, 2012) This page is in constant update and contains papers, tutorials, and links to other resources for clever experimentation on AMT. If you have any suggestion, write to gpaolacci-at-rsm.nl.
Tutorials
Screening participants from previous studies on Amazon Mechanical Turk and Qualtrics
Eyal Peer (Carnegie Mellon University), epeer@andrew.cmu.edu
Gabriele Paolacci (Erasmus University)
Jesse Chandler (Princeton University)
Pam Mueller (Princeton University)
Emailing workers using Python
Pam Mueller (Princeton University), pamuelle@princeton.edu
Jesse Chandler (Princeton University)
Fundamental articles on experimenting on AMT
Population Characteristics:
- Berinsky, A. J., Huber, & Lenz, G. S. (in press). Using Mechanical Turk as a Subject Recruitment Tool for Experimental Research. Political Analysis.
- Goodman, J. K., Cryder, C. E., & Cheema, A. (in press). Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making.
Data Quality:
- Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.
- Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are Your Participants Gaming the System? Screening Mechanical Turk workers. In Proceedings of the 28th international conference on Human factors in computing systems (2399-2402). New York: ACM.
- Horton, J.J., Rand, D.G., & Zeckhauser, R.J. (2011). The Online Laboratory: Conducting Experiments in a Real Labor Market. Experimental Economics, 4, 399-42
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
Instructional Guides:
- Chandler, J.J., Mueller, P., & Paolacci, G. (2011). Methodological Concerns and Advanced Uses of Amazon Mechanical Turk in Psychological Research. Working Paper.
- Mason, W. & Suri, S. (in press). Conducting behavioral research on Amazon’s Mechanical Turk. Behavioral Research Methods.
Group Dynamics:
- Suri, S., & Watts, D. J. (2011). A Study of Cooperation and Contagion in Web-Based, Networked Public Goods Experiments, PLoS ONE 6(3).
Online resources
- Sid Suri and Winter Mason maintain a wiki about research on AMT and other crowdsourcing platforms.
- Michael Buhrmester maintains a page about conducting experiments on AMT.
- Panos Ipeirotis’ blog is often posting about AMT.
- Dave Rand gave an introductory talk about conducting experiments on AMT.
- Adam Finkelstein’s course “Crowdsourcing your experiment” refers to many useful resources to run experiments on AMT.
- Deneme is another blog on experiments on AMT, with a different purpose. It also provides a link to Turkit, a Java/JavaScript API for running iterative tasks on AMT.
