The fight against misinformation has been crucial in recent years, with the Covid crisis literally making it a matter of life and death. Platforms have traditionally been deploying a mixture of AI-based tools and human moderators to try and weed out misinformation and prevent it from spreading.
Research from MIT suggests that laypeople can play a crucial role and proved as effective as professional fact-checkers in spotting fake news.
“One problem with fact-checking is that there is just way too much content for professional fact-checkers to be able to cover, especially within a reasonable time frame,” the researchers say.
Using the crowd
After examining around 200 stories that had been flagged by Facebook’s misinformation algorithm for additional scrutiny, the researchers found that a small and politically balanced group of laypeople could do a good job of evaluating the stories just from their headline and lead sentences.
“We found it to be encouraging,” the researchers say. “The average rating of a crowd of 10 to 15 people correlated as well with the fact-checkers’ judgments as the fact-checkers correlated with each other. This helps with the scalability problem because these raters were regular people without fact-checking training, and they just read the headlines and lead sentences without spending the time to do any research.”
This suggests that this crowdsourced approach is one that could be deployed widely and pretty cheaply, which has understandable attraction for platforms struggling to cope under the load. Indeed, the authors suggest that the cost of having readers evaluate stories works out around $0.90 per story.
“There’s no one thing that solves the problem of false news online,” the authors explain. “But we’re working to add promising approaches to the anti-misinformation tool kit.”
Wisdom of the crowds
The researchers believe the laypeople were able to have such success due to the classic phenomenon known as the wisdom of crowds. This posits that across a wide range of disciplines, laypeople have been shown to match or even exceed the performance levels of experts, due in large part to their diversity of opinions and ability to aggregate them.
The knowledge of the volunteers did seem to matter, however, with those who scored higher in a political knowledge test gravitating towards the thinking displayed by the professional fact-checkers.
“People that engaged in more reasoning and were more knowledgeable agreed more with the fact-checkers,” the researchers say. “And that was true regardless of whether they were Democrats or Republicans.”
This kind of approach is already being tested, with Facebook deploying a program called Community Review, in which laypeople assess news stories. Similarly, Twitter has the Birdwatch project, in which users are asked their input on various tweets.
It’s likely that such official efforts are going to be more effective than a free-for-all that is open to everyone, as such an approach may expose projects to more partisan influence. It’s a situation the researchers accept is largely unknown thus far.
“We haven’t yet tested this in an environment where anyone can opt in,” they explain. “Platforms shouldn’t necessarily expect that other crowdsourcing strategies would produce equally positive results.”
Given the scale of the challenge, however, it seems inevitable that social media platforms will have to seriously consider ways in which they can recruit a sufficient pool of laypeople to help in the task of news evaluation.
“Most people don’t care about politics and care enough to try to influence things,” the authors conclude. “But the concern is that if you let people rate any content they want, then the only people doing it will be the ones who want to game the system. Still, to me, a bigger concern than being swamped by zealots is the problem that no one would do it. It is a classic public goods problem: Society at large benefits from people identifying misinformation, but why should users bother to invest the time and effort to give ratings?”
