Honesty on AMT

(Guest post by Sid Suri)

For an online labor market such as AMT to function, there must be some degree of honesty between the workers and employers. Workers need to have some confidence that employers will pay them for their work. Employers need to have some confidence that workers will submit honest work.

In a series of three behavioral experiments, we measured the degree to which workers on AMT are honest and explore different factors that could affect their honesty. The first experiment in the series asked workers to roll a 6-sided die and report the outcome. If the worker didn’t have a die, a link to random.org was provided which simulates fair dice rolls. We paid the worker $0.25 plus $0.25 times the reported outcome, providing an obvious incentive to be dishonest. If everyone reported the roll honestly, the mean reported roll would be 3.5. The mean reported roll by the participants was 3.91, caused by an under reporting of 1’s and 2’s and an over reporting of 5’s and 6’s (see Figure 1 below). This is a clear indication of dishonesty, although perhaps less blatant than one might expect. One possible reason why the participants were as dishonest as we observed could be related to the relative amount they stood to gain by misreporting their roll.

Our second experiment aimed to test this hypothesis by changing the relative amount they could earn by misreporting their roll. We kept the average payoff about the same as in the first experiment but reduced the variance; we paid $1.00 plus $0.05 times the roll of a die. The results were not statistically different from the previous experiment, suggesting the variance in the pay was not the leading factor in our participants’ dishonesty.

Our final study explored how the detectability of deception would affect the behavior. When only rolling one die, it is impossible to know if a specific worker was being deceptive. Therefore, we asked workers to roll 30 dice and input all of the outcomes, so that we could at least detect egregious deception from a single individual. We paid the workers $0.01 times the sum of the rolls, and found the mean reported roll was 3.57—which, although still reliably different from fair, represents much less deception than the previous two studies. As Figures 1 and 2 (below) show there is a dramatic difference in the distribution of reported outcomes. One explanation for this result is that people may have some notion of what a likely distribution of rolls would look like, which guided them to cheat less and to cheat less egregiously in the multiple roll experiment so that they could not be caught. Finally, despite recording responses from a demographically diverse group of participants, we did not see any significant relationship between any of the reported characteristics and the probability of being dishonest.

Figure 1: The distribution of rolls in the single roll, baseline experiment. Error bars are confidence intervals for two- sided binomial tests relative to chance (p = 0.167) with Bonferroni-corrected α = 0.05.

Figure 2: The distribution of rolls in the multiple roll experiment (when each participant rolled 30 dice). Error bars are confidence intervals for two-sided binomial tests relative to chance (p = 0.167) with α = 0.05.

References

Suri, S., Goldstein, D., and Mason, W. (2011). Honesty in an Online Labor Market. Human Computation Workshop (HComp) 2011 (pdf).