AI and Knowledge Work

Early insights on the impact of AI on knowledge work

This is a weekly newsletter about the business of the technology industry. To receive Tanay’s Newsletter in your inbox, subscribe here for free:

---

Hi friends,

Happy NVIDIA earnings day to those who celebrate. It’s still early innings in terms of most knowledge workers using LLM and AI regularly in their jobs, but we’re starting to see some datapoints and anecdotes of its impact, particularly in areas such as customer support and software engineering where it has gotten adoption more quickly. 

Uber’s CEO, Dara Khosrowshi on the This Week in Startups podcast recently called out Uber’s use of AI for developer productivity and customer service:

So I think one one no brainer wins is developer productivity. So we now have a subset of our developers who are power users of GitHub Copilot, and it is excellent. And so the job now is to to and it's and it's truly adding productivity, but we now have to sell it from, like, 20% of those power users to 50% to 80%. It's it will be a home run for everybody. And it will take some of the kind of BS work away from developers so they can truly be creative. So I do think it's a win. The next for us is customer service… A human being has to go through all that. Now essentially we've taken the steps… First step is AI summarizes all of it … [and] gives the recommendation to the customer service agent.

This week, I’ll be discussing some of the early learnings about the impact of AI on knowledge work, based on studies and data across various fields.

---

Software Engineering

Paul Graham recently tweeted that one of the big benefits of AI is that it’s upskilling earlier-career developers who are more comfortable with AI to be as productive as more senior ones. 

While this is just anecdotal, we know that programmers through tools such as Github copilot and Cursor have been one of the earliest to adopt LLMs in their work. 

Github’s studies found that Copilot had an impact on:

• Speed: Github’s research showed developers were completing tasks 55% faster

• Confidence: 85% of developers felt more confident about their code quality

• Enjoyment: 88% of developers found it easier to maintain flow state

In fact, Microsoft has pointed to Copilot writing over 40% of code among users of it. While some of that includes code that may have been written by typeaheads and similar anyway, that’s still a staggering number!

---

Customer Support

In asking Fortune 500s how they plan on using AI, their use of AI to augment (and in more extreme cases deflect tickets/replace agents) in customer support probably comes up more often than any other area. 

An MIT and Stanford study from about mid 2023 called “Generative AI at Work”, which in some ways had a quite rudimentary setup, gave us a taste of what the impact of AI could be in this setting.

In the study, a generative chatbot was introduced which was trained on data from over 5,000 agents and their tickets, and was used to provide real-time suggestions for how to respond to customers to support agents.

The findings were:

• Improved efficiency: On average, support agents were able to resolve ~14% more tickets per hour, at the same satisfaction rate as before.

• Uneven impact: The impact varied widely based on worker tenure/skill level. The lowest skilled agents were 34% more productive using AI, whereas the most skilled ones had no improvements in productivity (in this setup).

• Faster learning: Being able to use AI allows the less tenured workers to move up the experience curve more quickly, and “mimic” in productivity their more tenured colleagues, as in the graph below.

  

• More empathetic communications: Since the AI suggestions aimed to also help agents be more empathetic with frustrated customers, customers treated the agents far more positively.

---

Consulting

Perhaps the most interesting study was a study by HBS on BCG consultants, where 750 consultants were put into 3 groups: no AI access, GPT-4 access, and GPT-4 access with training in prompting, and given a wide variety of tasks that they often do as part of their work.

The findings were:

• Improvements in productivity: Consultants using AI completed more tasks and more quickly. They completed 12.2% more tasks on average, and completed tasks 25.1% more quickly on average.

• Improvements in quality: They also completed these tasks at a higher level of quality, as much as 40% higher, as in the graph below (based on human and AI ratings of quality). 

  

  Source: Ethan Mollick

• Helped lower-performing consultants more: Consultants across the skills distribution benefited significantly from access to the AI, but those that were below average performance improved on average 43%, while those above average improved 17%. This is similar to the study above in customer support.

• AI is not for every task: The participants were also given tasks that are known to be ones outside the current capabilities of AI. For those tasks, those using AI performed worse, with them being 19% less likely to get to the correct answer / complete the task, compared to those who didn’t use AI. 

---

Closing Thoughts

Even though the interfaces and tools are still early, we’re starting to see some common themes of how they will impact knowledge work:

• Improve productivity and quality: They can help improve the productivity in terms of the number of tasks that someone can complete and also the quality of their output, as some of the studies alluded to.

• Uneven benefits: At least today, the benefits are quite uneven in some ways — those that are lower performers have more to gain by using AI, and it can help them ramp up to be performing at levels as top performers more quickly. But that doesn’t mean that it can’t help top performers as well. In addition to improving their productivity albeit to a smaller extent, it can also help them take out some of the drudgery from their work.

• Not a silver bullet: AI cannot help with everything yet, and know what it can and can’t do is quite important. Trying to use it to assist in tasks it’s not good at can actually lead to worse performance, as highlighted in the example of the BCG study. This will change over time as the capabilities improve, but it’ll take using AI regularly to develop an understanding of what to use it for assistance for vs not.