Posted by: Gabriele Paolacci | October 9, 2012

Slides from ACR 2012

On October 5, we held a special session at the Association for Consumer Research North American Conference called “Inside the Turk: Methodological Concerns and Solutions in Mechanical Turk Experimentation.” Below you can find the presenters’ slides (click on the title of the talks).

Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples (Joseph Goodman)
We compare Mechanical Turk participants to community and student samples on personality, financial, and consumption dimensions, as well as classic decision-making biases. We find many similarities between Mechanical Turk participants and traditional samples, but also find important differences researchers should consider when using Mechanical Turk for consumer research.

Screening Participants on Mechanical Turk: Techniques and Justifications (Emily Peel)
Concerns about the quality of Mechanical Turk participants induce researchers to screen participants. We evaluate screening strategies according to their discriminant ability to identify observations that contribute only noise. Our results suggest omitting participants based on these indicators would likely bias the sample rather than improve data quality.

Under the Radar: Determinants of Honesty in an Online Labor Market (Dan Goldstein)
Online subject pools depend on participants’ honesty. After establishing a baseline level of dishonesty on Mechanical Turk, we manipulate the incentives to cheat and the probability of detection. We find workers act like intuitive statisticians, cheating at a level below statistical detection at the individual, but not aggregate, level.

Non-Naivety among Experimental Participants on Amazon Mechanical Turk (Gabriele Paolacci, including introduction to the session)
We conducted two studies to identify the extent to which participant cross-talk and duplicate participation contribute to non-naivety among participants in Mechanical Turk. Whereas cross-talk is not a critical issue, there is evidence of numerous duplicate participants. We discuss the implications for Mechanical Turk experimentation.





  1. Very useful, thanks for sharing that.

  2. Hey, thanks for the ressources, but is it just me or the first slides are corrupted (from Goodman)?

  3. All files are safe and working.

  4. […] Slides from ACR 2012 (good tips) […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: