The last few years have seen researchers use the crowd extensively in their work, whether it’s using games like Phylo to perform genetic research, the partnership between Twitter and Gnip to provide our tweets to researchers, or even the use of platforms such as Minecraft for research purposes.

Another fertile area is to use platforms such as Mechanical Turk for what are known as ‘human intelligence tasks’ (HIT).  The size and scale of this marketplace was unveiled in a recent report published by the Pew Research Center.

Human intelligence

The paper reveals that around 800 studies were published in 2015 using data from Mechanical Turk, and the paper aimed to delve into this burgeoning market to discover what kind of researchers use it, why they use it, and what kind of ‘subjects’ the studies attract.

Central to the analysis was data from the Mechanical Turk Tracker, a data repository created by Dr. Panagiotis G. Ipeirotis of the New York University Stern School of Business.

So who is responding to research based requests on the site?  Interestingly, whilst the image is typically one of lowly-skilled individuals being exploited for low pay, the analysis actually revealed that the typical participant was more likely to have a degree than the norm.

That isn’t to say that they aren’t being exploited however, with the average pay nearly $2.50 below the federal minimum wage in the US.  Perhaps unsurprisingly therefore, many participants simply use Mechanical Turk to top up their primary source of income, although worryingly some 25% used it as their primary source.

Whilst this income tended to be lower for older ‘Turkers’, it was usually younger people that relied exclusively on the platform for their income.

What kind of research?

During the period of analysis, it emerged that academics were responsible for around 36% of all tasks posted to the site.  So what kind of things are they requesting?

By far the most common were small, repetitive tasks that could be performed by users in a few minutes, with the bounty being around 10 cents.  The tasks were usually things that are easy for humans but tough for machines, such as identifying information in an image or the completion of a survey.

Whilst business users of the site tended to post multiple requests, it was more common for academics to post once, and once only.  Of the commercial users, they represented a relatively small number of high-frequency users, with the platform therefore a significant part of their business model.

As such, the academic community appear to be using the platform as an alternative to large, perhaps more expensive, surveys of the general public.  There is continued debate as to the merits of this, not least from Princeton University, with studies exploring how researchers should sample users on the platform to accurately reflect the wider public.

There is also a concern that so bombarded are Turkers with research that they have become highly skilled at completing surveys.  Researchers have expressed concern that such familiarity might flavor the results.

Whilst it can be hard to fathom the motivation behind participation for such low remuneration, the study does suggest that most users do so on a frequent basis.

It seems likely that Mechanical Turk will be used by researchers for some time to come, and this report provides a nice insight into just who is using the platform, and some of the challenges that researchers face with it.