Using MTurk to Estimate Effects of Local Context in Behavioral Games

Guest Post by Mark Keuschnigg

Local confinement of samples and results has motivated questions as to the external validity of social science experiments. The last 15 years have thus seen a sharp increase in experiments conducted at multiple locations including developing countries and small-scale societies. However, cross-regional comparisons of economic behavior have run into obstacles due to limited transferability of standardized decision situations into parallel laboratory set-ups. In a recent article in Social Science Research we utilize cross-regional participation at MTurk to circumvent common pitfalls of traditional multi-location experimentation.

We argue that MTurk experiments provide a sorely needed complement to laboratory research, transporting a homogeneous decision situation into various living conditions and social contexts. In fact, we believe that quasi-experimental variation of the characteristics people bring to the experimental situation is the key potential of crowdsourced online designs. Our research shows that such analyses of “virtual pools” can be adapted to study local patterns of behavior.

We use Ultimatum (UG) and Dictator Games (DG) for data generation (N = 991). We use bargaining games specifically because norms of fairness are strongly conditional on local context. UG and DG thus reveal expectations about valid social norms in a particular population.

To assess the importance of context, our design includes an experimental variation of monetary stakes ($0, $1, $4, and $10) as a benchmark. Our marginal totals correspond closely to laboratory findings: Monetary incentives induce more selfish behavior but, in line with most laboratory findings, the particular size of a positive stake appears irrelevant.

Analyses of “virtual pools” first mirror standard sub-group analyses contrasting participants from different regions. We illustrate this by comparing workers’ behavior from India and the US: Controlling for differences in the socio-demographic composition of national pools we do not find a cross-country difference in a parametric situation (DG). Culture, however, seems to be relevant in strategic interaction (UG). Participants in India are more selfish (proposers) and less demanding (responders) than US Americans. Within the US, Southerners appeared both more selfish (proposers) and more demanding (responders) than Northerners.

More importantly, however, participants’ geographical locations provide an interface for direct inclusion of macro variables potentially influencing individual behavior. We limit our analysis to regional variation of economic affluence and social capital across US states. According to our estimates, dictators’ allocations from wealthier and socially more integrated states are 13 percent larger on average than those from less-advantaged states. Interestingly, total size of contextual influence clearly exceeds stake effects, and most important from a sociological perspective, context effects are both more pronounced and theoretically consistent than effects of individual attributes.

• For cross-country comparability we used tokens and weighted payoffs for Indian participants using a purchasing power parity conversion factor.
• To balance national pools we posted four HITs daily (early morning and late afternoon at local time in each country) and recruited for each daily session as many US Americans as we had recruited Indians earlier that day.
• To avoid waiting time and drop-out, actual matching of subjects occurred only before payoff from a pool of preceding participants’ decisions (without replacement).
• Submissions were only accepted once per worker-ID. We also disabled participation from IP-addresses similar to those existing in our database to impede multiple participations by one household.

So far, the use of “virtual pools” has received scant attention in experimental research. We know a great deal about how institutional arrangements affect fairness, trust, cooperation, and reciprocity in economic games, yet we know little about how local socio-economic conditions and strategies learned in daily interaction influence outcomes of social experiments. Bringing context back into social experiments is particularly relevant for sociological research which—unlike most experimental research in economics and psychology—fully acknowledges the importance of context effects in a multi-level explanation of individual action.

Reference

Keuschnigg, M., Bader, F., Bracher, J. (2016) Using crowdsourced online experiments to study context-dependency of behavior. Social Science Research, doi: 10.1016/j.ssresearch.2016.04.014.