Turkers of the world, unite!

Amazon Mechanical Turk (MTurk) is an online platform for crowdsourced labor used extensively by computer scientists to gather data and test software products as well as by behavioral scientists to conduct scientific experiments. Requesters on MTurk design isolated "micro-tasks" called Human Intelligence Tasks (HITs) and post requests for workers to complete these tasks for prespecified piece work wages. Amazon presents MTurk as a kind of impersonal black box for micro-task labor, and requesters then naturally conceive of the platform as such. Therefore, it is easy for the requesters to fall into the trap of assuming that MTurk is akin to a kind of idealized "frictionless" world of "spherical chickens", where the responses of workers are free from any effects of the social contexts of the workers.

ILLUSTRATION BY: LAURENT HRYBYKCREATIVE ART DIRECTION: REDINDHI STUDIO

Recent academic work on the MTurk community itself has attempted to raise this thin veil and highlight to researchers the humanity of the people themselves engaged in crowdwork on MTurk, and the implications of the fact that Turkers are not cogs in a distributed human computer or a psychology data simulator.

We contribute to this growing literature by showing that even if workers do not communicate directly, the fact that they belong to a coherent community of MTurk workers produces an in-group bias that can affect the results of certain types of tasks on MTurk. Our findings are meant to draw attention to the inevitable contextual factors at play in the social contexts of MTurk workers. Keeping these contextual factors in mind can aid in the interpretation of scientific experiments and user studies conducted on MTurk, and especially in interpreting the generalizability of findings from these enterprises.

Our results confirm that being a "Turker" is a strong enough identity to have impact on experiments conducted on MTurk. We use the "Dictator Game" wherein participants decide how to split a $1 bonus between themselves and an anonymous counterpart of varying group identities. Our results indicate preferential in-group bias by MTurk crowdworkers, where donations to other workers on MTurk were 15% higher than those given to workers from another platform (i.e., CrowdFlower), and 35% higher than the ones given to a random person. As such, findings from studies conducted on MTurk about collaboration, cooperation, or competition would then potentially generalize most directly to environments where people share an identity associated with their workplace.

Related and selected content:

  • Working Paper: Almaatouq, Abdullah and Krafft, Peter and Dunham, Yarrow and Rand, David G. and Pentland, Alex, Turkers of the World Unite: Multilevel In-Group Bias Amongst Crowdworkers on Amazon Mechanical Turk (February 12, 2018). Available at SSRN: https://ssrn.com/abstract=3122520