Given the hype surrounding generative AI since ChatGPT burst onto the scene, one could be forgiven for thinking it’s the panacea to all of our woes. Indeed, so pronounced are its capabilities said to be that many have argued of a looming destruction of jobs (again).
As research from Harvard demonstrates, however, the reality is that generative AI is likely to be helpful in some tasks, and harmful in others.
To the test
The researchers worked with over 750 consultants from the Boston Consulting Group, each of whom was randomly assigned to work on a task that was designed to be as realistic as their day-to-day work as possible. One of the tasks was more analytic in nature, and asked the consultants to analyze the performance of a company, before providing the CEO with recommendations.
The second task was much more creative, with the consultants tasked with developing a new footwear product to serve an as-yet underserved segment of the market for a fashion brand.
The researchers divided the consultants into three groups. The first group used AI without any instructions. The second group used AI after watching a short training video. The third group had no access to AI.
AI support
Those using AI finished both tasks faster. However, the quality of their work varied greatly between the two tasks. In the creative footwear task, those using AI outperformed those without it by 40%. The lower-performing consultants saw the biggest improvements, catching up with their stronger peers.
But in the strategic decision-making task, those using ChatGPT-4 did worse than those without AI. AI users were 20% less likely to find correct solutions. Still, their recommendations were rated higher because they were more persuasive and well-written.
In short, consultants using AI were more likely to be wrong but sounded more convincing. This creates a significant issue for businesses.
Where the limits lie
The results highlight one of the core challenges with generative AI. Given the extreme amount of hype surrounding the technology, it’s extremely difficult to know where its capabilities end. Indeed, Sam Altman has even tried to claim that hallucinations are a feature rather than a flaw.
You can trust AI to help with tasks within its “jagged frontier” and get high-quality results. But if you use it for tasks beyond this frontier, you’re more likely to make mistakes.
This is tricky because Large Language Models (LLMs) are still fundamentally opaque. Sometimes they produce incorrect results that look plausible and convincing, making it hard to predict their failures.
Even if you accurately identify AI’s current limits, the technology is evolving so quickly that these boundaries could shift tomorrow.
Decreasing diversity
Another challenge posed when people use generative AI en masse is that it can result in a reduction in cognitive diversity. The researchers noticed that the pool of creative ideas became smaller in the group using generative AI to help them.
In the footwear task, for instance, those using AI often came up with strikingly similar ideas. Consultants without AI worked more slowly and their ideas were generally of lower quality. However, they produced a more diverse range of proposals.
This suggests that companies could benefit from relying more on human creativity to generate unique ideas. This is especially important when radical innovation is needed and competitors heavily use AI.
There is no simple answer to the question, “For which tasks should companies use AI?” Instead, businesses should experiment systematically and strategically with different uses.
Finding your way
The study is a timely reminder that the best approach is not to blithely take the marketing hype at its word and assume that generative AI will be the answer to all of your problems, and instead employ a more experimental approach whereby it is tested to see where it helps and where it doesn’t.
Your aim should be to gain a proper understanding of where the technology might be best suited and also understand the various risks and unintended consequences associated with it.
If you simply deploy AI haphazardly it could reduce productivity, diminish human creativity, and harm the accountability that is essential to the most important tasks.
Of course, this isn’t to say that AI shouldn’t be used, as ignoring it could also have consequences in terms of your competitive edge. Merely that there are likely to be instances where the technology helps, and instances where it doesn’t. It’s your job to experiment so you fully understand which is which.
The best way to use AI will remain uncertain for the near future. Therefore, managers will need to keep experimenting with the technology as it evolves.
Even if companies don’t question whether to adopt AI, they should consider how to use it responsibly to make work more productive and meaningful for employees working at the jagged frontier of their abilities.
