Slides from ACR 2012

On October 5, we held a special session at the Association for Consumer Research North American Conference called “Inside the Turk: Methodological Concerns and Solutions in Mechanical Turk Experimentation.” Below you can find the presenters’ slides (click on the title of the talks).

Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples (Joseph Goodman)
We compare Mechanical Turk participants to community and student samples on personality, financial, and consumption dimensions, as well as classic decision-making biases. We find many similarities between Mechanical Turk participants and traditional samples, but also find important differences researchers should consider when using Mechanical Turk for consumer research.

Screening Participants on Mechanical Turk: Techniques and Justifications (Emily Peel)
Concerns about the quality of Mechanical Turk participants induce researchers to screen participants. We evaluate screening strategies according to their discriminant ability to identify observations that contribute only noise. Our results suggest omitting participants based on these indicators would likely bias the sample rather than improve data quality.

Under the Radar: Determinants of Honesty in an Online Labor Market (Dan Goldstein)
Online subject pools depend on participants’ honesty. After establishing a baseline level of dishonesty on Mechanical Turk, we manipulate the incentives to cheat and the probability of detection. We find workers act like intuitive statisticians, cheating at a level below statistical detection at the individual, but not aggregate, level.

Non-Naivety among Experimental Participants on Amazon Mechanical Turk (Gabriele Paolacci, including introduction to the session)
We conducted two studies to identify the extent to which participant cross-talk and duplicate participation contribute to non-naivety among participants in Mechanical Turk. Whereas cross-talk is not a critical issue, there is evidence of numerous duplicate participants. We discuss the implications for Mechanical Turk experimentation.