The ROI on Generative AI

A few anecdotes of real, tangible ROI on AI in production from enterprises

This is a weekly newsletter about the business of the technology industry. To receive Tanay’s Newsletter in your inbox, subscribe here for free:

---

Hi friends,

The AI landscape is marked by stark contrasts. While some experts debate how soon we'll achieve Artificial General Intelligence, others remain skeptical of AI's current real-world impact. Despite massive investments across the AI stack and the acknowledged overbuilding in AI we touched on a few weeks ago, critics argue that tangible results often seem elusive, and it seems like everyone is just experimenting with it for the sake of it within enterprises.

This week, I’ll discuss some of the actual, tangible ROI that large enterprises are seeing from Generative AI, based on what they shared on their earnings calls¹.

---

1. Customer Support

This is probably one of the areas we’re seeing the most actual uptake and early results from Generative AI.

The poster child for this has been Klarna which leveraged AI to elevate their customer support. Their AI assistant has taken over the work of 700 employees, reducing resolution times from 11 minutes to just 2 minutes while maintaining high customer satisfaction levels.

"In the first half of 2024, we saw firsthand the benefits of adopting practical AI. We also made a conscious decision to address the societal impacts of generative AI. Our AI assistant now performs the work of 700 employees, reducing the average resolution time from 11 minutes to just 2, while maintaining the same customer satisfaction scores as human agents.

We're excited to share the adoption and tangible benefits the next evolution of our AI assistant brings. Our AI assistant now also serves as a powerful shopping assistant that helps consumers discover and choose products tailored to them, further transforming the shopping experience and helping them save time and money.

Salesforce’s Agentforce platform, which helps resolve tickets is another one that is showing real ROI for early customers.

What we saw with Wiley was, this is a quote from them, 'we're seeing double-digit percentage increase in customer satisfaction and deflection rate compared to older technologies and in these early weeks of our busiest season.'

So that was very reassuring to us. that we have the right thing that's happening. And Wiley has already seen a 50% increase in case resolution. That's with our first generation of Agentforce, as I mentioned, the second generation of Agentforce, which we have with customers already, including some of these amazing organizations like Royal Bank of Canada, ADP and others is 90% case resolution. It is an awesome moment in this tech business.

Microsoft casually dropped that they too are expecting to save hundreds of millions of dollars a year on call centers after adopting Generative AI.

[Microsoft] Dynamics is another exciting place for us, which is one, we are gaining share. Dynamics with GenAI built-in is sort of really a biz app, it's probably the category that gets completely transformed with GenAI. Contact centers being a great example. We ourselves are on course to save hundreds of millions of dollars in our own customer support and contact center operations. I think we can drive that value to our customers. - Satya Nadella

---

2. Software Engineering

Software engineering has been another area of heavy adoption, with some even going on to say that engineers days are numbered. While I don’t think that’s true, the signs of it radically increasing productivity are already clear among those who have learned to use the latest AI tools such as Cursor, GitHub Copilot and leveraging Claude’s Sonnet. But we’re also hearing examples of measurable, tangible benefits from enterprises, as Amazon shared about their software development assistant Q which has saved them over 4,500 developer years in a recent code migration task.

"We’re continuing to see strong adoption of Amazon Q, the most capable generative AI-powered assistant for software development and to leverage your own data…

… It also saves development team's time and money on the muck nobody likes to talk about. For instance, when companies decide to upgrade from one version of a framework to another, it takes development teams many months, sometimes years burning valuable opportunity costs and churning developers who hate this tedious though important work.

With Q's code transformation capabilities, Amazon has migrated over 30,000 Java JDK applications in a few months, saving the company $260 million and 4,500 developer years compared to what it would have otherwise cost. That's the game changer. And think about how this Q transformation capability might evolve to address other elusive but highly desired migrations."

---

3. Customer Experience

Generative AI is also being used to alter the customer experience, either improving it radically or changing the economics to serve customers.

eBay launched a new AI-assisted selling flow, and are already seeing improvements in customer satisfaction as well as faster time to list and get value for Sellers.

The success of the German initiative influenced our decision to invest in an improved experience for preowned apparel in the U.K. last quarter. These changes, including a new selling flow that makes it considerably easier to list on eBay with image guidance, simpler item aspect collection and a streamlined shipping module with a focused set of couriers and parameters

For users of the new selling flow, we have observed a 20-point increase in CSAT, a material reduction in average listing time and an increase in listings and sold items per seller. A growing number of buyers have engaged with our new AI-powered fashion shopping features and are eagerly awaiting the enhanced personalization and filtering we plan to bring to Shop the Look and Explore in the coming months.

YUM Brands is enhancing customer experiences at Taco Bell by rolling out voice AI driven drive-through systems. This technology is not only improving customer satisfaction but also boosting team member productivity, and the results are so promising they are accelerating their roll-out timelines.

"As you recall, last quarter, we discussed plans to expand drive-through Voice AI technology to more Taco Bell stores. I'm excited to announce that given our encouraging early results, the team has accelerated the rollout. And as of today, we now have this technology operational in over 100 Taco Bell stores. We plan to scale this technology to several hundred stores by year-end, while the pilot test is underway in KFC Australia. In our tests, we have witnessed consistent consumer experiences and higher team member productivity."

---

If you don’t yet receive Tanay's newsletter in your email inbox, please join the 9,000+ subscribers who do:

---

4. Sales

In startup circles, sales is another area of both many companies seeing pull, as well as adoption to improve GTM motions. While there are worries of mass spam only increasing via cold outbound and cold calling powered by AI, AI also has a clear role to play in helping AEs be better AEs (and at some point being an AE itself).

Manulife is an example of a company already seeing large ROI in using AI to assist salespeople. The AI is used to assist salespeople to recommend better products to prospective and current clients (insurance, wealth management, retirement planning) and personalize the retirement planning process for its clients. They are seeing promising results and expect double digit increases in repurchase rates from customers.

So to recap … is that we're using GenAI and machine learning models to make it really easy for agents to understand customer opportunities but also to generate these personalized communications at the click of a button to help them engage with more customers more often.

In our first 2 weeks live, about 68% of our agents had already used the new GenAI capabilities. And in July, we will be broadening that user base to about 2,000.

Based on our analysis in Singapore, we anticipate a 17% uplift and repurchase rates for our customer base, when this is fully rolled out to all of our agents.

It's still early days. So it's difficult to get a full picture of where this will go. But at a minimum, we expect to see about a 5% increase in APE sales impact across our businesses."

---

5. Operational Processes

We have models that are now good across many different domains, and in particular are very effective when used for digitizing and automating what would be paper processes and tedious data entry type grunt work, and we’re already seeing companies see large ROI from this.

Rocket Mortgage is utilizing AI to automating the transcription of client calls and completing mortgage applications. This has drastically improved banker productivity and ensured accuracy in high-stakes transactions, saving bankers from having to input tens of millions of data fields each week.

"The next example is the rollout of Rocket Logic Assistant to our entire banking force, helping our clients navigate the home-buying journey. This AI-powered personal assistant transcribes client calls and automatically completes mortgage applications in real time, supercharging our banker productivity.

Gone are the days of manual notetaking with hands on keyboard or pen and paper, which would fatigue our bankers and leave gaps in client conversations.

Now the Rocket Logic Assistant seamlessly generates over 300,000 detailed transcripts every week from outbound calls. It supports over 100 data points on mortgage applications saving our bankers from inputting tens of millions of data fields each week.

This enhances efficiency allows us to closely monitor calls and extracts valuable client insights while also creating recursive models and feedback loops to continuously improve our bankers performance and more effectively train each new generation of the best bankers in the country."

Walmart has harnessed generative AI to enhance its product catalog, improving the quality of over 850 million pieces of data. This initiative has significantly reduced the manual effort required and enhanced the customer experience by providing more accurate and helpful product information.

"We're finding tangible ways to leverage generative AI to improve the customer, member and associate experience. We're leveraging data and large language models from others and building our own.

One example is that we've used generative AI to improve our product catalog. The quality of the data in our catalog affects nearly everything we do, from helping customers find and buy what they're looking for, to how we sort inventory in the network, to delivering orders. We've used multiple large language models to accurately create or improve over 850 million pieces of data in the catalog. Without the use of generative AI, this work would have required nearly 100x the current head count to complete in the same amount of time. And for associates picking online orders, showing them high-quality images of product packaging helps them quickly find what they're looking for."

Lastly, the models are also helping make sense of unstructured data and provide gains in areas where traditional ML has already been used in the past. Mastercard is leveraging the new advances in Generative AI to enhance fraud detection, achieving a 20% increase in accuracy.

"So AI has been massively transforming to us and allowed us to generate a tremendous amount of value. Now in, new set of techniques. We have what we're now calling Decision Intelligence, when I talked about earlier, DI Pro. We're using recurrent neural networks and other generative techniques to just like ChatGPT works by guessing the next word and they're generating incredibly compelling spicy text. We can now look at anonymized payment histories of any actor in the network, and that's what's happening next in the way we could. So it's taking the attention spreading it wider and getting much more fidelity. So again, we've seen a 20% increase in fraud detection on top of all the amazing things we had already been able to do with AI. So that's why I'm saying when we talked a little bit about it earlier, it's not a new game, but it's an amazing new club to have in the bag to apply to those use cases where it makes best."

---

Closing Thoughts

While it is true that many workloads are still experimental, there are clear signs that workloads are starting to hit production and show real value. From enhancing customer support and engineering processes to improving sales strategies and operational workflows, AI is proving to be a powerful tool for driving efficiency and real ROI at companies across a wide range of functions and areas.

As more companies figure out the 2-3 areas out of the many pilots they may be running with AI where there is tangible value, I expect to see many more diverse examples of real ROI. And we’re still so early as the models are only getting better and the cost of intelligence is only decreasing.

1. All quotes are sourced directly from the earnings call of the specific company