I’ve written previously about the use of the crowd to help fight legal battles, but most of these efforts have revolved around crowdfunding money to help your case.

A recent paper suggests that the crowd can also be invaluable in understanding whether a case will win or not.  The paper examined FantasySCOTUS, an online fantasy league in which players predict the outcome of cases at the US Supreme Court, with the best performers winning a range of prizes.

Since the competition began in 2011, over 7,000 players have competed, making over 600,000 predictions between them on around 400 different cases.

Good judges?

So how good were the crowd at predicting the outcome of cases?  The research suggests that they’re actually pretty good.  The team devised a wide range of models to do their own predictions based upon the data from the game.  In total, something like 250,000 different models were created.

“We exhaustively simulate models over a wide range of reasonable choices, and then analyze the performance of these models in aggregate,” the authors say.

These models were compared with a null model, which was the heuristic that lawyers tend to use when guessing how cases will unfold.  It basically suggests that the Supreme Court will usually reverse the decision of any lower court, which is what happens around 60% of the time.

Despite having so many different models in the mix, the crowd outperformed them all.

“We provide strong support for the claim that crowdsourcing can accurately and robustly predict the decisions of the Supreme Court of the United States,” they explain.

Suffice to say, they aren’t the only team using interesting methods at predicting legal cases.  A few years ago a team of researchers developed an AI system to do the same thing.

The researchers, from UCL, the University of Pennsylvania and the University of Sheffield, developed a system that could predict the decisions made by the European Court of Human Rights (ECtHR) with an accuracy of 79%.

The system is believed to be the first of its kind in the world, and arrived at it’s verdict simply by absorbing the text in the case files.

“We don’t see AI replacing judges or lawyers, but we think they’d find it useful for rapidly identifying patterns in cases that lead to certain outcomes. It could also be a valuable tool for highlighting which cases are most likely to be violations of the European Convention on Human Rights,” they explain.

Whether it’s AI or the crowd, it seems that we are likely to get much better at predicting the outcome of legal proceedings.