We’re pretty used to having AI play a role in the recruitment process, with many employers using technology to filter through the hundreds or thousands of applicants they receive for each vacancy. In such circumstances, there are obvious concerns about the potential for biases to be introduced, especially if the systems are trained on historical data that has inevitable biases (although research suggests women actually think AI systems might be fairer).

Could AI be used for other forms of evaluation? That was the question posed by a recent study from Texas McCombs, which looked at whether technology could effectively evaluate business plans.

Under the microscope

Submitting business plans to banks, VCs, and even our internal managers is a key part of the process of making our ideas reality. The people analyzing those pitches often believe themselves to be experts at the task, and apply their expertise to determine whether an idea is worth backing or not.

The study suggests that not only is AI a match for these so-called experts, but it can often outperform them in terms of selecting the best projects. What’s more, it can also help to spot ideas that might otherwise have fallen into a blind spot.

The researchers were inspired by the success of AI in the trading world, with algorithmic traders now doing the vast majority of trades. While AI has shown some promise in the automation of strategy (particularly in the private equity business), it’s a domain that is still dominated by humans. Indeed, it’s often an echo of Jeremy Irons’ character in the movie Margin Call, who famously said that his only real job was to understand “when the music might stop”.

Supplanting experts

The researchers teamed up with a startup accelerator in Europe. They analyzed 10 business plans, half of which had been accepted and half that had been rejected.

They fed each of the plans into ChatGPT and asked the system to create an AI double. The prompt contained language from the original business plan, which always described the problem the entrepreneurs were looking to tackle. The technology was then left to create the remainder of the business plan on its own.

The resultant business plans were then shown to a pool of 250 evaluators, each of whom had an average of around five years’ experience in investing, and around seven years’ in management. They reviewed a combination of AI-generated business plans and human-generated business plans (but, obviously, never the AI generated doubles of the human originals).

The evaluations were conducted according to the key aspects of each business plan, such as the value proposition, the level of innovation, the viability of the project, and the investment potential. Each attribute was scored from 1-10 before a final judgment was offered as to whether they would back the project financially, whether it should be allowed into the accelerator, or whether they would like a meeting with the founders.

Worryingly, the evaluators consistently rated the business plans written by AI to be better. What’s more, they were rated higher across all of the criteria. For instance, the AI-produced plans were  5% more likely to be accepted into the accelerator and 3% more likely to be offered investment.

Picking the winners

The researchers then flipped the tables, and tasked AI with assessing the business plans rather than writing them. This time, the researchers gathered submissions to a startup competition hosted by an elite business school. They collected 138 business plans in total, each of which had already been evaluated by the human judges, most of whom were investors and venture capitalists.

The AI was given the same kind of prompts given to the human judges to try and replicate the conditions reliably. The business plans were also put through three separate AI assessments to try and mimic the real life experience.

As in the previous experiment, the AI performed admirably, and was generally producing recommendations similar to that of the experienced investors. Indeed, they tended to correlate more towards the view of the most experienced investor than the human judges, for whom there was a wider spread.

“We were quite surprised with the correlation we got,” the researchers explain. “AI was quite accurate in picking the winners, and it was also better at predicting which plan was ultimately going to be successful.”

Doing strategy

The results were so promising that the authors believe AI could play a crucial role in the strategic planning process. For instance, VCs could use AI to evaluate business plans; accelerators could use it to evaluate applications; and entrepreneurs themselves could use it to create and test their business plans.

“Everyone now has access to high-level strategic decision-making,” the researchers explain. “It’s McKinsey-in-a-box. It democratizes it.”

As such, it’s perhaps not surprising that they’re bullish about the prospects of AI being incorporated into the strategic planning process to give people a tangible strategic advantage (until everyone does so, obviously).

“This is going to lead to new strategy frameworks being developed,” they conclude. “It will change strategy decision-making, not just in the way that we do it, but in the way that we perceive it.”