Posted by: Gabriele Paolacci | May 11, 2012

Can AMT be Used for Learning and Memory Experiments?

Guest post by Todd Gureckis and John McDonnell

AMT has been used in behavioral research for a variety of purposes including norming stimuli, simple judgement and decision making tasks, or collective behavior experiments. While previous work has suggested that AMT data compares well to data collected in the laboratory, fewer studies have looked specifically at the types tasks that might interest many cognitive scientists.  For example, we are not aware of work addressing tasks that involve learning a non-trivial concept over a number of trials. There is good reason to suspect that AMT may not be ideal for such tasks because they typically require sustained focus, 20-60 minutes of time, and careful consideration of the task instructions.
We explored this issue by replicating a classic study in cognitive science on human concept learning (Shepard, Hovland, & Jenkins, 1961).  Key features of this study include that it has been widely replicated in different labs and depends on learning and problem solving extending over many trials.
Our first attempt at replication did not result in the same qualitative pattern of effects found reliably in laboratory replications. However, in a sequence of follow-up studies we explored the variables that effect the quality of data on AMT. In our first follow-up, we explored the impact that participant incentives have on the quality of AMT data. We found that payment had little effect on performance or the quality of data and mostly effected the signup rate.
However, in our final experiment we did something entirely obvious but perhaps a little insidious. We had participants answer a simple questionnaire at the end of the instruction phase of the experiment that tested for comprehension of the key details of the study. If the participant did not answer perfectly we had them return to the instruction phase.  This repeated in an infinite loop until participants could master all the questions. After this simple no-cost manipulation our data became much more similar to previous laboratory reports. Many participants repeated the instruction phase more than three times.
Based on this experience, we offer three lessons to researchers interested in using AMT in their work.  First, it is important to verify that subject understand the instructions (obvious, but we didn’t realize how important that could be).  Second, like other reports we found that the magnitude of payment doesn’t effect the overall quality of the data. Third, we found that it is possible to conduct learning and memory experiments that result in data that is very similar to that obtained in laboratory settings.

Read about our full results here.


Shepard, R., Hovland, C. L., & Jenkins, H. M. (1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75(13), 1–42.



  1. While your research may have indicated that pay does not affect quality of data I would hope you would consider that those doing the work are trying to earn money for very real reasons (food, shelter) and would not lower the pay rate simply because you can. It may not increase the quality of your data to pay at least minimum wage (calculated by time required) however it does enhance the ethics of your project.

  2. Absolutely. It is important to know whether and how pay rates affect data quality, but this doesn’t imply data quality is the only thing that matters when deciding on pay rates. Requesters should always be considerate and transparent in their choice of wage/effort required. Thanks for your comment!

  3. Eris: I absolutely agree with this point. In fact, the point at the end of our blog post is that ethical considerations should lead researchers to pay roughly in line with laboratory rates (taking into account there is more hassle involved in actually coming into the lab to be a participant in terms of travel, etc…).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: