In previous articles, I’ve explained that a lot of the productivity challenges countries face are due to the unequal dissemination of new technologies. In other words, those (often bigger) companies at the vanguard capitalize on technologies like AI and robotics, while laggards (who are often smaller) do not.

Is this same heuristic true in the world of generative AI? That was the question posed by a recent study from Berkeley Haas, which explored how the technology was being used by small businesses in Kenya.

Meaningful impact

The study found there was a decidedly mixed picture for small businesses attempting to utilize generative AI. While the most successful were able to secure a performance boost of around 15%, those at the lower end actually saw their revenue fall by 8%.

When they dived into these figures to understand the discrepancy, it emerged that the higher performers tended to use generative AI for fairly simple and straightforward tasks. By contrast, those companies that weren’t getting good returns were asking much more challenging things of the technology.

“Generative AI has the potential to significantly influence business performance,” the researchers explain. “Our results suggest that whether this impact is positive or negative depends on the tasks that entrepreneurs select for AI assistance.”

Tech to the test

During a five-month randomized control trial, 640 Kenyan entrepreneurs, operating various ventures such as fast food joints, poultry farms, and cybershops, were examined to understand their technology adoption levels. The study aimed to assess the effectiveness of a GPT-4-powered AI business “mentor” that had been developed by the researchers. This AI interacted with entrepreneurs through WhatsApp, a widely used messaging platform in Kenya.

Prior to the trial, the researchers dedicated months to tailoring the AI to the specific nuances of the Kenyan business landscape. They programmed it to offer practical advice and detailed implementation strategies for a range of business scenarios. During the trial, participating entrepreneurs were randomly assigned to either receive guidance from the AI or follow a standard business guide.

Each entrepreneur was free to consult the AI mentor as needed, seeking advice on various aspects of their businesses. For instance, a restaurant owner deliberating a menu change sought assistance in weighing the options and potential outcomes. Similarly, another entrepreneur involved in wholesale and retail milk sales sought guidance on expanding product offerings to boost profits.

Minimal change

Over a five-month study period, the researchers collected around 4,000 different data points to gauge company performance, with the firms conducting thousands of interactions with their AI mentor.

On the whole, they found no real positive impact from using the AI mentor in terms of firm performance. When they dived into the data more closely, however, a clear split emerged, with some companies deriving value from the technology and others seeing a fall in their performance after using it.

The researchers discovered that while there was no disparity in the quantity or overall caliber of questions posed by the two groups, there were notable distinctions in the nature of the inquiries. Entrepreneurs categorized as low performers, already grappling with diminished revenues and profits, tended to seek guidance on complex challenges that posed significant hurdles for both AI and human advisors.

These challenges included scenarios such as fierce market competition, adverse weather conditions like droughts impacting farms, or urgent financial needs to sustain business operations.

“Whether by choice or necessity, low-performing entrepreneurs in our sample asked the AI mentor for assistance with more challenging tasks than high performers,” the researchers explain.

A leveling force

The findings are interesting, as early research into how generative AI was being used suggests that it could have a leveling impact, in terms of improving the performance of less capable people to that of their more talented peers.

The results from this study perhaps support that conclusion in that the most successful applications of the technology were for more straightforward queries.

As such, despite the somewhat modest findings from the study, the researchers are fairly bullish about the potential for the technology to have a significant impact on smaller businesses that may traditionally have been left behind by the latest technologies.

“This suggests that for gen AI to really add value to entrepreneurs in more open-ended contexts, they would also need expanded access to complementary skills training and resources—including financial resources,” the researchers conclude.

“An optimistic way to view our results is that we had a positive effect for a subset of the population, with a very low-cost intervention.”