We organized a Mechanical Turk Special Session at the Association for Consumer Research North American Conference (October 4-7 2012, Vancouver). The session is called “Inside the Turk: Methodological Concerns and Solutions in Mechanical Turk Experimentation” and will take place on Friday, October 5 at 2pm.
Whereas initial evaluations of AMT as a source of experimental data have emphasized its compelling strengths, fewer efforts have been made to explore and quantify its potential unique drawbacks and limitations. The special session will focus on some of the issues that threaten experimental validity on AMT and on providing easily implementable solutions to avoid these problems. The four papers included in the session deal with diverse issues. Joe Goodman will discuss differences between AMT workers and more traditional subject populations that are of high relevance to consumer behavior research. Julie Downs will discuss strategies for restricting data collection and data retention to attentive participants, together with their implications for the generalizability of AMT data. Dan Goldstein will address issues of participant honesty, including the results of experiments designed to detect dishonest behaviors among AMT participants and identify some of their predictors. Gabriele Paolacci will address the issue of non-naïvety among AMT workers by presenting studies about cross-talk and duplicate participation and provide simple remedies to attenuate this concern.
We plan to share our slides on the blog after the conference, so stay tuned!