Guest post by Neil Stewart
How many people can your lab reach on MTurk? We used the capture-recapture method¹ from wildlife ecology to estimate how many workers you are sampling from. Our estimate is 7,300 workers.
Using 114,460 HITs completed from 2012 onwards we estimated, for each of our labs, how many workers we are sampling from. We then used a random-effect meta-analysis to estimate the number of workers a new lab, which could be yours, could reach. Why does this matter? Well, there is an exponential-like increase in the number of publications using this MTurk population—and we, like others, have found considerable overlap between our laboratories. And if you are planning a series of experiments or running adequately powered replications, you could run out of workers quite fast.
What can you do to increase your reach? Surprisingly, paying more doesn’t help. Our population estimate was reduced for higher paying HITs—we think because the most active workers seek out these HITs and crowd out the less active workers. (Still, no reason not to pay a living wage to our participants!) Running larger batch sizes does help. Our larger batches sampled from a population nearly three times larger than the smaller batches. One last strategy is to wait. We estimate that it takes about 7 months for half of the workers on MTurk to leave and be replaced.
View the paper in-press in Judgment and Decision Making here.
Neil Stewart, Christoph Ungemach, Adam Harris, Dan Bartels, Ben Newell, Gabriele Paolacci, and Jesse Chandler
¹ The intuition behind capture-recapture method is not too hard. Ecologists might, for example, use it to estimate the number of fish in a pond. Go fishing on Day 1. Catch some fish, tag them, and release them. Then, on Day 2, go fishing again. Catch some fish and observed the proportion that are tagged. Now you have an estimate of the proportion tagged in the pond from Day 2, and the number tagged in the pond from Day 1, so you can estimate the total number in the pond. If you tag five fish on Day 1 and observe one quarter are tagged in Day 2’s catch then there must be 20 fish. We used WorkerIDs like tags.
Experimental Turk 
[…] The Average Laboratory Samples a Population of 7,300 AmazonMechanical Turk Workers (JDM, 2015) (Summary post on Experimental Turk) […]
By: Running Experiments with Amazon Mechanical Turk | Gilad Feldman on July 30, 2015
at 7:34 pm
I think it’s important to mention the qualifications those HITs were using if the labs were targeting specific geographic locations. It wasn’t the entire AMT workforce you had access to if you were using the location qualification. My reach on my surveys has been ~3,600 in 24 hours using no qualifications at all, while I was able to reach 30,000 with my mapping HIT over approximately 45 days (but it took real effort.)
By: spamgirl on July 31, 2015
at 2:50 am
Turkprime has tools that could help get more varied results across experiments. For example you can exclude anyone from a previous HIT (or HITs) from taking the new one. This would restrict the more active users on MTurk from being in multiple experiments.
By: Ben on March 11, 2016
at 6:50 pm