Customer service agents using an AI tool to guide their conversations saw productivity gains of nearly 14%, with 35% improvements for low-skilled and least experienced workers, and zero or small negative impact on more experienced/skilled workers, Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond report in Generative AI at Work (NBER Working Paper 31161).
Using call data from approximately 5,000 agents working for a Fortune 500 software company, researchers tracked the duration, quality and outcome of customer service interactions as the company introduced a GPT AI tool (Generative Pre-trained Transformers). The tool was rolled out to agents gradually, mostly between November 2020 and February 2021. For a control group, the researchers also collected data from agents who did not receive the tool during 2020 and 2021. The AI tool had the purpose of supporting the work of human customer support agents, offering them potential answers to customer questions. Agents can choose to take those suggestions or ignore them and enter their own responses.
With AI assistance, customer service agents could handle more calls per hour and increase their resolution rate.
Researchers found that customer service agents using the AI tool increased the number of customer issues resolved per hour by 13.8%. They attribute the increase to three factors: Agents, who could participate in multiple chats simultaneously, spent about 9% less time per chat, handled about 14% more chats per hour, and successfully resolved about ‘1.3% more chat overall. Customer satisfaction measures showed no significant changes, suggesting that productivity improvements did not come at the expense of interaction quality.
The researchers split the data by agent tenure length and pre-AI productivity, and find that the benefits of using the AI tool were greatest among less experienced and less skilled workers, who saw gains of 35%, with little or no adverse effects on top performing/most experienced workers. An agent using the AI tool who only had a two-month tenure with the company performed as well as an agent with a six-month tenure working without the tool. The researchers suggest that newer, less skilled workers may have more to learn than more skilled and more established workers, and that AI tools can help them adopt the skills and behavior of more experienced workers more quickly. Text analysis of agent conversations supports this interpretation.
All agents changed their communication patterns after they started using the AI tool, but the change among the underperforming agents was greatest. This may be because the AI tool based its suggestions on the work style and performance of the company’s most productive agents, and then disseminated their behavior pattern to new and less skilled workers. For example, the developers of the AI tool found that the top performing companies were able to determine the underlying technical problem, based on a customer’s description, twice as fast as the least performing companies. The AI tool, trained using the best examples of resolved queries, learned to connect specific query phrases to helpful diagnostic questions and potential solutions. The AI tool was also able to provide more frequent feedback than a human manager. This gave new hires and underperforming employees the opportunity to improve faster than they would have without the tool, by iterating with each call rather than just following managerial reviews.
The researchers also noted that clients were more likely to express positive feelings and less likely to request help from a supervisor when interacting with agents who used AI assistance than when interacting with those who didn’t. Perhaps reflecting improved trading tenor, attrition rates among agents with access to the AI tool were 8.6% lower than comparable rates for agents without such access.
Emma Solomon
#Measuring #productivity #impact #generative