AMT has been used in behavioral research for a variety of purposes including norming stimuli, simple judgement and decision making tasks, or collective behavior experiments. While previous work has suggested that AMT data compares well to data collected in the laboratory, fewer studies have looked specifically at the types tasks that might interest many cognitive scientists. For example, we are not aware of work addressing tasks that involve learning a non-trivial concept over a number of trials. There is good reason to suspect that AMT may not be ideal for such tasks because they typically require sustained focus, 20-60 minutes of time, and careful consideration of the task instructions.
We explored this issue by replicating a classic study in cognitive science on human concept learning (Shepard, Hovland, & Jenkins, 1961). Key features of this study include that it has been widely replicated in different labs and depends on learning and problem solving extending over many trials.
Our first attempt at replication did not result in the same qualitative pattern of effects found reliably in laboratory replications. However, in a sequence of follow-up studies we explored the variables that effect the quality of data on AMT. In our first follow-up, we explored the impact that participant incentives have on the quality of AMT data. We found that payment had little effect on performance or the quality of data and mostly effected the signup rate.
However, in our final experiment we did something entirely obvious but perhaps a little insidious. We had participants answer a simple questionnaire at the end of the instruction phase of the experiment that tested for comprehension of the key details of the study. If the participant did not answer perfectly we had them return to the instruction phase. This repeated in an infinite loop until participants could master all the questions. After this simple no-cost manipulation our data became much more similar to previous laboratory reports. Many participants repeated the instruction phase more than three times.
Based on this experience, we offer three lessons to researchers interested in using AMT in their work. First, it is important to verify that subject understand the instructions (obvious, but we didn’t realize how important that could be). Second, like other reports we found that the magnitude of payment doesn’t effect the overall quality of the data. Third, we found that it is possible to conduct learning and memory experiments that result in data that is very similar to that obtained in laboratory settings.
Read about our full results here.
Shepard, R., Hovland, C. L., & Jenkins, H. M. (1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75(13), 1–42.