Posted by: Gabriele Paolacci | July 11, 2013

How naïve are MTurk workers?

More and more social scientists have adopted MTurk as a venue for their research, praising its speed, cost, and diversity relative to undergraduate samples. However, many of them may fail to take into account some other critical aspects that differentiate MTurk samples from undergraduate subject pool samples.

In a paper just published in Behavior Research Methods, we find worker non-naïveté to be a serious concern.  One general issue is that MTurk workers share information about HITs with each other publicly and searchably on various forums, including on two different subreddits (see here and here for some collected examples of manipulation checks and common measures that have become common knowledge among workers via forum).

More specifically, while the probability that any worker has seen some manipulation may be low, there is a population of “superturkers”, i.e., extremely prolific workers, who are significantly more likely to end up in your studies. We pooled 16,408 HITs in 132 unique studies, and found that the HITs were completed by 7,498 unique workers.  While the average worker completed 2.2 HITs, the top 1% of most prolific workers (15+ HITs) completed 11% of the total, and the top 10% (5+ HITs) completed nearly half (41%) of the total HITs.

superturkers

As mentioned, superturkers can be problematic because:

  • They are more likely to have seen standard manipulations (see figure below)
  • They are significantly more likely to read MTurk blogs/forums
  • They are significantly more likely to receive notifications from www.turkalert.com each time you (as an academic requester) post new HITs

exposure

However, on the plus side, we find that:

  • They are less likely to be multitasking while on MTurk
  • They are much better at responding and completing follow-up studies (one year later, 75% of these workers completed a follow-up), if you are interested in longitudinal research.

In sum, superturkers are a mixed blessing.  MTurk has the capability for more sophisticated designs, including longitudinal studies, and superturkers are reliable enough to make this viable.  Since they are less likely to be multitasking, they may also be good participants in studies that require more attention (e.g., reaction time).

However, their non-naïveté, and worker non-naïveté in general, is a serious concern.  Researchers can and should take steps to exclude workers from subsequent studies within their lines of research (the paper provides one solution; this method is another good solution if your studies are on Qualtrics).  Ideally, they would also communicate with other researchers who do similar work, so that one’s previous participants could be excluded from the other’s study, and vice versa.  Beyond that, though, researchers should be very wary about using “classic” manipulations or measures on MTurk.  The next blog post will detail the data that supports this admonition.

References

Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior Research Methods, 46(1), 112-130.

About these ads

Responses

  1. And college sophomores (who constitute the largest source of subjects in social science studies) never do these things?

  2. The question is if they do it more than once. Labs should take care of that, of course.

  3. […] are keen to mTurk recognize that it is not a panacea. We recognize that it has its limitations (see How naive are MTurk workers”), but accept those limitations as part of the research design itself and either work within the […]

  4. […] a previous post, we documented the existence of MTurk workers that are disproportionately likely to show up in […]

  5. Nice post. I like the way you start and then conclude your thoughts. Thanks for this information….

  6. 楽天メンズファッション ミズノ アシックス http://chuanqirut.onlinecentury2013.com/

  7. […] How naïve are MTurk workers? and the followup response – mTurk: Method, Not Panacea and the followup post – Consequences of Worker Nonnaïvete: The Cognitive Reflection Test […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

Join 33 other followers

%d bloggers like this: