The CFO Edit | How CFOs Should Actually Measure AI ROI
12 Nov, 20267
AI investment is growing, but how many businesses can confidently say what they're getting in return?
For CFOs, the challenge is no longer simply deciding where to adopt AI, but understanding whether that investment is making a measurable difference to the business. Lower model costs don't necessarily mean lower AI spending, and widespread adoption doesn't automatically translate into better productivity.
In this edition of The CFO Edit, we speak with Conor O'Donoghue , CFO of Ocrolus , about what it takes to turn AI experimentation into measurable business value. From managing rising compute costs to rethinking how teams measure productivity, Conor shares the lessons he's learned and what CFOs should be looking at when assessing AI's return on investment.
In the early days of generative AI, companies rushed to experiment. As CFO of a scaling fintech, what did that initial adoption curve look like from your seat?
We jumped headfirst into it. Machine learning and automation have always been central to what we do at Ocrolus, so there was already a strong appetite to explore how generative AI could improve our operations and products.
Adoption happened across three areas: our core back end technology, the customer facing platform, and internal workflows. Early on, we invested in building an internal calibration model, trained on our own proprietary data, which dynamically determines the best model to use for each task and context. This helps us get the right result without automatically defaulting to the most expensive model.
One of the interesting things we've seen is while the cost per unit of AI has come down considerably, overall spending hasn't necessarily followed the same direction. Ramp reported that the cost per million tokens across its vendor base had fallen by roughly 80%, yet aggregate AI spending remained steady. That demonstrates how quickly usage is growing.
When did AI experimentation start becoming a problem for the budget?
It became noticeable within the first three to six months. We started seeing how quickly costs could build up when people used powerful models for relatively simple tasks.
One example was using Claude’s latest model to sort and triage emails. At around $3 per email, it was an expensive way to handle something that had minimal productivity gains and could be done with a much cheaper tool.
It was a useful wake-up call. Not every task needs the most advanced model available, and businesses need to think more carefully about which tools they're using, what they're paying for, and how to set rules on how the models get used.
The way you measure AI usage also matters. If you focus on token consumption, you risk encouraging people to use more tokens rather than achieve better business outcomes. The focus needs to be on what the technology is helping the business achieve, rather than how much of it people are using.
How should CFOs measure AI ROI beyond tracking usage and token consumption?
Token usage and licence costs tell you how much AI a business is consuming, but they don't tell you whether the business is performing better. I think the more useful approach is to look at the impact within each department and identify the metrics that should improve.
In Engineering, for example, you can look at pull requests, development cycle times, and deployment speed. We're seeing the potential for small, cross-functional engineering pods, bringing together product managers, engineers and subject matter experts, to ship features in days or weeks rather than waiting for traditional development sprints. That's a much more meaningful measure of productivity than simply counting how many people have access to an AI coding assistant.
In Sales and Marketing, you might look at quota attainment per Account Executive, pipeline generated per salesperson, and the ratio of Sales Development Representatives to Account Executives. In Customer Success, it's worth looking at whether Account Managers can support more active accounts without affecting Net Revenue Retention.
For G&A teams, you could measure how quickly the finance team closes the books, how long vendor onboarding takes, or how many complex client contracts the legal team can review each week without bringing in outside counsel.
The important question is whether teams can achieve more with the resources they already have. If AI is helping a business improve productivity, reduce costs or grow without increasing headcount at the same pace, that's where you start to see a return.
What are the biggest pitfalls businesses should watch out for when introducing AI into their operations?
One of the biggest things to understand is that AI models are probabilistic. They don't always produce the same answer, which can become a real problem when you're dealing with finance, compliance, or anything that requires a precise result. A model that does a reconciliation correctly 95% of the time is no use to a CFO, because finance needs the same answer every time.
You need to be thoughtful about where you use AI and where you still need deterministic processes. For example, rather than asking an AI model to carry out a financial reconciliation itself, you can ask it to generate the Python script or Excel logic that performs the calculations. The model helps build the solution, while the underlying maths is handled deterministically. You still need the right checks and human oversight, particularly when the output has financial or compliance implications.
Another challenge is giving employees too much freedom to build their own tools. People can spend hours experimenting with prompts and creating side projects, which can lead to duplicated work and systems that aren't properly controlled. We've seen Account Executives spend time tinkering with prompts and building side projects when that time could be spent speaking to prospects. At the end of the day we want employees spending the majority of their time where they can provide the most value to the company using their unique skillset that can’t be replicated better by other areas of the business or AI.
You can also end up with plenty of data and insights that don't actually lead to any change. We built customer sentiment dashboards that produced useful reports, but if nobody changes their workflows or takes action based on the findings, the technology isn't delivering value. If AI tooling isn't shortening cycle times or reducing the need for additional back-office headcount, it risks becoming shelfware.
How do you make AI part of the company culture rather than just another tool employees experiment with?
I think it starts with making sure people have access to the right tools and a clear understanding of how they're expected to use them. If everyone is experimenting independently, it's difficult to build consistent workflows or understand what's actually working.
It also means thinking about how roles and performance expectations might change. Individual contributors may be able to take on more execution, managers can spend more time coordinating work, and senior leaders can focus on how the business scales. We need to consider how AI changes what good performance looks like at each level.
It changes how we hire. Domain knowledge still matters, but we also look at whether someone can use AI to get more done. An engineer or analyst who knows how to work with these tools can often outperform a team of three working the old way.
It changes how we judge performance too. If someone spends hours on a manual process we could automate, that isn’t diligence any more. We’d see it as something to fix.
It's also important to think beyond the immediate task. For example, if someone is building an automated bank reconciliation, don't ask them to create something that only works for October's Chase bank accounts. Build something that can work across all bank accounts and regions. That's where you start to get repeatable value rather than a collection of one-off solutions.
I also see AI as a useful thought partner. You can ask it to suggest ten possible ways forward. Five might be completely irrelevant, three might be fairly obvious, but two could give you a genuinely useful angle that you can adapt to your business. It's about knowing how to challenge the output and use it to improve your own thinking, rather than accepting everything it gives you.
Looking forward, what is your advice to fellow CFOs grappling with AI ROI in their upcoming planning cycles?
Treat AI like any other business investment. Be curious about what it can do, but be disciplined about where the money goes and what you expect to get back.
I'd be cautious about accepting claims around hours saved or productivity improvements without understanding what those gains mean for the business. Are costs coming down? Can the company grow without hiring at the same pace? Are teams delivering more, or are they simply using more tools?
Every meaningful investment should have a business owner and a metric attached to it. If you can't explain what you're trying to improve or how you'll measure the result, it's difficult to know whether the investment is worthwhile.
The technology is moving quickly, and there's a real opportunity for businesses that get it right. But the CFO's role is to make sure that adoption translates into something tangible.
If you missed our previous editions of The CFO Edit, you can catch up on the latest three here:
Related Posts
The CFO Edit | Why Executives Confuse Risk with Fear
16 Sept, 2026
