Guest Post by Danielle Shapiro and Jesse Chandler
Relative to behavioral sciences, clinical sciences have been slow to adopt MTurk as a recruitment tool. This is unfortunate because a major obstacle of clinical research is locating individuals who score in the extremes of clinically relevant variables, who by definition make up the minority of the population and are thus hard to find in large numbers.
We investigated the use of MTurk as a recruitment tool for populations of interest to clinical scientists. In line with numerous previous studies of MTurk, we found that data quality was high. Scale items used to measure underlying psychological constructs held together well. More importantly, the relationship between self-reported demographic information, life experiences, and psychological constructs was largely consistent with prior research, e.g., unemployment predicts depression, women report more anxiety, and men drink more alcoholic beverages. In general, workers looked a lot like the US population as a whole, except they reported surprisingly high levels of social anxiety.
We also learned a few things that may be of interest to researchers in other fields. First, in line with previous results, we found that workers are basically honest about personal details when payment is not contingent on their responses. We asked workers to report demographic information at two different time points more than a week apart, and for workers from US IP addresses, virtually all of them reported the same information both times.
Second, workers may be less honest when details relevant to payment are concerned. For example, a surprising number (around 6%) of workers who claimed US residence in fact came from IP addresses assigned to Eastern Europe and India. This is probably because US based workers are paid in cash rather than Amazon credit. Similarly, we measured malingering – the tendency to report symptoms that seem clinically relevant but are in fact rarely reported in clinical populations. We found a substantial portion of the population (around 10%) reported unusually high levels (>3 SD above the norm) of malingering. One interpretation of this finding is that workers infer the purpose of a survey and try to provide information that is relevant to what the requester wants. This interpretation is in line with earlier research that shows a higher level of social desirability bias among MTurk workers than among other populations (Behrend, Sharek, Meade & Wiebe, 2011).
Moreover, we learned that a substantial proportion of workers are unemployed or underemployed – far more than the US national average. The extent to which these workers are willing to work for very low wages should not be construed as satisfaction with current payment rates.
You can find our full report here.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813
Shapiro, D.N., Chandler, J., & Mueller, P. (in press). Using Mechanical Turk to Study Clinical Populations. Clinical Psychological Science.