Experimenting on AMT: Fundamental Articles

We put together a list of articles that covers the most critical aspects of experimentation on AMT. You can find the list below organized by theme. Most likely the list will be updated in the future, and you will always find the most recent version in the Resources page of the blog.

In the next weeks, we plan to start publishing some tutorials: step-by-step explanations of how to perform certain operations on AMT, e.g., mass emailing workers or screening out non-naive participants. Contributions are most welcome.

Population Characteristics:

• Berinsky, A. J., Huber, & Lenz, G. S. (in press). Using Mechanical Turk as a Subject Recruitment Tool for Experimental Research. Political Analysis.

• Goodman, J. K., Cryder, C. E., & Cheema, A. (in press). Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making.

Data Quality:

• Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

• Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are Your Participants Gaming the System? Screening Mechanical Turk workers. In Proceedings of the 28th international conference on Human factors in computing systems (2399-2402). New York: ACM.

• Horton, J.J., Rand, D.G., & Zeckhauser, R.J. (2011). The Online Laboratory: Conducting Experiments in a Real Labor Market. Experimental Economics, 4, 399-42

• Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.

Instructional Guides:

• Chandler, J.J., Mueller, P., & Paolacci, G. (2011). Methodological Concerns and Advanced Uses of Amazon Mechanical Turk in Psychological Research. Working Paper.

• Mason, W. & Suri, S. (in press). Conducting behavioral research on Amazon’s Mechanical Turk. Behavioral Research Methods.

Group Dynamics:

• Suri, S., & Watts, D. J. (2011). A Study of Cooperation and Contagion in Web-Based, Networked Public Goods Experiments, PLoS ONE 6(3).