A recent study conducted by MIT researchers has provided valuable insights into the effects of generative AI technology on work, dispelling some of the exaggerated claims surrounding its capabilities. The study revealed that when applied to tasks such as composing cover letters, delicate emails, and cost-benefit analyses, generative AI resulted in increased productivity among workers.
While the assigned tasks in the study did not demand precise factual accuracy or intricate knowledge of specific contextual details, participants noted similarities between the tasks and their real-world job responsibilities. The advantages of utilizing the assistive chatbot, ChatGPT, were significant. Workers experienced a remarkable 40 percent reduction in task completion time, while the quality of their output, as evaluated by independent experts, improved by 18 percent.
By making their findings available in the open-access journal Science, the researchers aim to enhance understanding of the real-world implications of AI tools like ChatGPT and their potential impact on the workforce. This study contributes valuable insights into the tangible benefits that AI can offer in improving work efficiency and output quality.
Raising productivity
Researchers conducted a study involving 453 college-educated professionals, including marketers, grant writers, consultants, data analysts, human resource professionals, and managers, to assess the impact of generative AI on worker productivity.
Participants were assigned two occupation-specific writing tasks, such as crafting cover letters for grant applications, composing emails on organizational restructuring, and developing customer analysis plans based on provided data. Their submissions were evaluated by experienced professionals in their respective fields, who were unaware of which submissions involved assistance from ChatGPT, an AI chatbot developed by OpenAI.
During the second task, half of the participants were granted access to ChatGPT-3.5. The users utilizing the chatbot completed their tasks 11 minutes faster than the control group, and their average quality evaluations increased by 18 percent.
Reduced variance
The data analysis also revealed a reduction in performance inequality among workers. Those who received lower grades in the first task derived greater benefits from leveraging ChatGPT for the second task.
While the researchers acknowledged several limitations, such as the inability to demand contextual knowledge about specific companies or customers due to anonymous participants, explicit instructions for each assignment, and the absence of fact-checkers to verify output accuracy, they maintained that the tasks reasonably represented real-world assignments faced by professionals.
The researchers cautioned that these limitations could potentially diminish ChatGPT’s productivity-enhancing potential in practical settings. However, the study’s findings highlighted the promising nature of the technology. Notably, workers exposed to ChatGPT during the experiment were twice as likely to report using it in their actual job two weeks later, providing further support for its perceived value and efficacy.
“The experiment demonstrates that it does bring significant speed benefits, even if those speed benefits are lesser in the real world because you need to spend time fact-checking and writing the prompts,” the researchers conclude.
