How to Measure AI ROI When Most Enterprise Pilots Are Just Theater

To learn how to measure AI ROI, finance leaders should stop tracking vanity adoption metrics and tie specific AI workflows to hard operational outcomes such as cycle time, error rates, and cash balances, according to speakers from Atlassian and Google Cloud. Mercedes-Benz reduced an eight-month code refactoring process to eight days by focusing on a specific operational bottleneck.

Sven Peters at Atlassian recalled tracking company-wide super users who engaged with AI 40 times a week. The company doubled that usage within six months, but the spike came almost entirely from engineering while legal and marketing abandoned the tools.

Why do vanity metrics hide the real AI return?

Peters admitted that Atlassian set the wrong incentives and measurements by focusing on company-wide numbers. The team moved to tracking super users per function. He reported that only 6% of executives see a return on investment from AI.

Katrin Lehmann at Mercedes-Benz reported that over 50% of the workforce uses AI, with a goal of pushing daily usage past 70% within the next few months. Every Friday morning, the company blocks out time for the whole organization to drop the daily grind and learn, with experts running hands-on training so employees can build their own agents.

A company-wide adoption number can look great on a slide and still hide which departments use the tools and which gave up on them.

Should you chase many use cases or a few?

Francis deSouza at Google Cloud recommended picking five to seven use cases a year and driving them from the top down. Companies that let 1,000 flowers bloom, he warned, very often end up with 1,000 dead flowers.

One panelist estimated that off-the-shelf copilots deliver about 20% of AI's total value, and the other 80% comes from the heavy lifting of building custom models for internal business processes.

Peters said Atlassian found its hero use cases in practical places: call prep agents for sales, intelligent document processing for finance, and an HR agent for new employee onboarding. Together with other work, those freed up 27,000 hours. For scale, deSouza noted that over 50% of the code generated at Google is generated using AI.

What is a framework for how to measure AI ROI?

The teams seeing returns point AI at specific operational bottlenecks. deSouza recalled that Google's treasury department uses Gemini to optimize daily cash balances, and that move generated tens of millions of dollars in benefit.

Lehmann said Mercedes-Benz pointed AI at its legacy mainframe systems and cut a code refactoring effort that would have run eight months down to eight days. Peters said Atlassian aimed AI at pull request reviews and cut cycle time by 45%.

What are the hidden costs of AI?

One panelist recalled an early customer that burned through $120 million on coding agents alone. The customer found that stale knowledge in Confluence, Notion, and other systems sent agents down incorrect paths and wasted over half of its token spend.

Another speaker said their small media organization experimented with AI every day, buying a new subscription and realizing after 15 days that it had been a mistake. A third pointed out how unpredictable the math is: it is not obvious whether a prompt will create $10,000 worth of tokens or $100 worth.

One panelist laid out the numbers for physical machines. We spend $1.5 trillion a year maintaining critical machines, waste $150 billion of it on broken service operations, and lose another $100 billion annually on warranties and recalls. Parts are increasingly scarce, the speaker added, and that holds up 40% of those repairs.

What does automation cost people?

Alison Moore at Chief shared a statistic that should make every executive pause: 87% of women leaders have watched things go wrong when companies prioritize AI without investing equally in their people. She pointed to a chief product officer who lets AI work through customer feedback while the call on which product features to kill stays 100% human, because the trade-offs are too nuanced to automate. Moore said three in four people expect the critical thinking gap to widen by April 2029.

Peters recalled a colleague who asked AI to write an event strategy and then asked Peters for feedback before they had discussed it. Peters argued that people should use AI to challenge their thinking and keep their own thinking in the loop. His colleague's version cost time and trust.

If you want to know how to measure AI ROI without fooling yourself, the answer is plain: stop watching broad adoption numbers and track specific workflow improvements.

To hear more from the people putting AI into production, see the HumanX 2027 program preview.