Using MTurk to Study Political Ideology

Guest post by Scott Clifford, Ryan Jewell, and Philip Waggoner

MTurk is increasingly used to study questions about politics and political psychology. MTurk samples are well known to deviate from the national population on a number of dimensions, particularly political ideology. The underrepresentation of conservatives has led some scholars, most notably Dan Kahan, to worry that the conservatives who opt into MTurk are not “real” conservatives. As Dan puts it, the underrepresentation of conservatives means we “can infer there is something different about the conservatives who do sign up from the ones who don’t” (see Dan’s discussion here and here). For example, they may differ from other conservatives in psychological dispositions central to their identities. If this claim were true, it may render MTurk samples invalid for studying political and ideological divides. This would be particularly worrisome for research using ideology or partisanship as a moderator of experimental treatment effects or examining psychological differences between liberals and conservatives.

In a recent article published in Research & Politics, we evaluated this concern using a large sample recruited from MTurk (N = 1,500). We compared this sample to two nationally representative benchmark surveys from the American National Election Studies 2012 Time Series Study, which was conducted before and after the 2012 US presidential election. The ANES study recruited 1,413 respondents for face-to-face interviews and 3,860 respondents for a web-based survey (through GfK).

In our MTurk survey we asked a series of questions that allowed us to make a direct comparison to the ANES surveys. Following research in political psychology, we focused on two sets of variables that we expected would be associated with political ideology: the Big Five personality traits and values (egalitarianism, moral traditionalism, racial resentment, authoritarianism). Our first analysis consisted of looking at the levels of each trait and value across political ideology. Contrary to the concerns discussed above, our MTurk conservatives looked nearly identical to ANES conservatives across all of these measures. Surprisingly, it was liberals who looked different – our MTurk liberals consistently held more liberal patterns of values and issue attitudes than our ANES liberals.

As our primary test, we estimated models predicting political ideology as a function of either personality traits or values, while controlling for standard demographics. The figure below plots the coefficients and 95% confidence intervals for each sample. As is clear from the figure, the results are highly similar across samples. In fact, across a broader set of tests, over 90% of the coefficients are statistically indistinguishable in size across samples. Thus, a researcher investigating the psychological predictors of political ideology would have reached largely the same conclusions whether using MTurk or the ANES.

Overall, we found that liberals and conservatives closely mirror the psychological divisions of liberals and conservatives in the mass public, providing little support for the concern that self-selection creates a pool of conservatives who are psychologically distinct from their counterparts in the larger population. We did find, however, that MTurk liberals hold more characteristically liberal values and attitudes than liberals from nationally representative samples. As a result, we encourage researchers to use more robust measures of political ideology, such as an index of political attitudes, in order to more fully capture variation in political ideology. Nonetheless, we find little reason to believe that liberals and conservatives recruited from MTurk are psychologically distinct from their counterparts in the mass public.