It has become increasingly common for scientific studies that require large numbers of people for platforms such as Amazon’s Mechanical Turk to be used to recruit and manage participants, each of whom is paid a relatively small sum for their participation.

Whilst there may be ethical issues surrounding the recruitment of people for such paltry sums, a recent study from Penn State suggests that such microworkers may not even provide good value for money.

The study found that people recruited through Mechanical Turk were more likely to give minimal effort, whilst also experiencing moods that could impact the results of the study.

“Researchers don’t want participants to have superficial involvement in their studies because, as a result, the rigor of these studies, and therefore the findings, could be compromised,” the authors say.

Cognitive dissonance

The researchers believe that the low pay that is typically awarded Turkers for their work can introduce a level of cognitive dissonance to proceedings, as their actions are disconnected from their thoughts or beliefs.  This can occur when the compensation given doesn’t seem fair compared to the effort expected.

“People might handle cognitive dissonance for this in a few different ways,” the researchers say. “For example, if they experience cognitive dissonance about saving water, they might change their minds and say saving water isn’t that important, or they may try to align their actions with their beliefs and change their showering habits.”

Interestingly, those who experienced cognitive dissonance were most likely to take contrary positions.  For instance, participants who were lowly paid were much more likely to say that their performance was important to them than highly paid workers.  This in turn was found to cloud their perceptions during any research they were asked to carry out, which may in turn skew the results.

“Our data showed that overstating the importance of the study is a coping tactic because it is associated with more enjoyment and less tension in their minds while filling out the questionnaire,” the researchers say. “However, it is also linked to a tendency to please the researchers by providing answers that they think the researchers are looking for rather than honest, authentic responses.”

Poor results

What’s more, lowly paid microworkers also tended to drop out of studies earlier than expected, with less effort given to them as well, resulting in poorer quality outcomes for the research.

Despite this apparent weakness, the platforms are becoming increasingly popular Petri dishes for research, with hundreds of experiments being undertaken at any given time.  The researchers urge any academics using the platform to weigh up the benefits of an easy to access, round-the-clock labor force against the lower quality inputs they can provide.

The team plan to further investigate the microworking community to understand whether intrinsic or extrinsic motivations work best, and indeed the role cognitive dissonance plays in their willingness and ability to participate in future tasks.  With the research community increasingly turning to Turkers for volunteers then it’s crucial that they understand what they’re getting.