To what extent can researchers rely on AMT for experiments with simultaneous participation? Designs that require participants to interact with each other at the same time are very common (e.g., in experimental economics), yet it is difficult to implement them on the web: One needs to have a certain number of participants to be present at a certain time, to handle dropouts, etc. Although AMT has some features that might facilitate the implementation of synchronous designs, there are only a few available examples. Mason and Suri (2010) describe a possible procedure to run synchronous experiments on AMT, which was successfully used by Suri and Watts (2010). Another description of how a synchronous study was conducted on AMT can be found in Lydia Chilton’s Master’s Thesis. We welcome other examples of studies with simultaneous participation conducted in AMT (or elsewhere).
References
Chilton, Lydia B. (2009). Seaweed: A Web Application for Designing Economic Games. M.Eng. Thesis, Massachusetts Institute of Technology.
Mason, Winter and Suri, Siddharth, Conducting Behavioral Research on Amazon’s Mechanical Turk (October 12, 2010). Available at SSRN: http://ssrn.com/abstract=1691163
Suri, Siddharth and Watts, Duncan J. (2010). Cooperation and contagion in networked public goods experiments. Available at: http://arxiv.org/abs/ 1008.1276
A HCOMP 2009 paper called KissKissBan needed to do some synchronous ESP-game like tests and paired players on MTurk. I think their strategy was similar to mine – lure people with a high-paying hit, and have them wait around for 30 seconds or so before forming a partnership.
http://portal.acm.org/citation.cfm?id=1600153
I’ve also heard about inviting turkers to come back at a specific time to form a group. This can get larger groups together, in theory, but I haven’t tried it myself.
In general, I think synchronous tasks deserve a good technical solution – there is a demand for it.
By: Lydia Chilton on January 13, 2011
at 6:26 am