Do workers on AMT pay attention to the directions provided by experimenters?
We recruited participants from AMT, discussion boards around the Internet, and an introductory subject pool at a Midwestern American University. After completing some tasks, participants were submitted the Subjective Numeracy Scale (SNS; Fagerlin et al. 2007). The SNS is an eight-item self-report measure of perceived ability to perform various mathematical tasks and preference for the use of numerical versus prose information, and it provided an ideal context for an Instructional Manipulation Check (IMC), as discussed in Oppenheimer et al. (2009). Included with the SNS, participants read a question that required them to give a precise and obvious answer (“While watching the television, have you ever had a fatal heart attack?”). This question employed a six-point scale anchored on “Never” and “Often” very similar to those in the SNS, thus representing an ideal test of whether participants paid attention to the survey or not.
Participants in the three subject pools did not differ in terms of attention provided to the survey. Participants in Mechanical Turk had the lowest IMC failing rate, although the number of respondents who failed the IMC is very low and not significantly different across subject pools, χ2(2, 301) = .187, p = 0.91. See here for a more detailed analysis of the experiment.
Fagerlin, A., Zikmund-Fisher, B., Ubel, P., Jankovic, A., Derry, H., & Smith, D. (2007). Measuring numeracy without a math test: Development of the subjective numeracy scale. Medical Decision Making, 27, 672–680.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.