Measuring the ROI of AI without inventing numbers
A baseline, two or three metrics, a second measurement: how to judge an AI project on facts rather than on estimated savings.
Thibault Pons de Vincent
Founder of Oliverlist and VivaSync
AI pitches usually come with a savings figure: so many hours saved a week, so many more files handled. Many of those figures are calculated before the tool has run for a single day. If you report to a board, to partners or to family shareholders, sooner or later someone will ask you for the ROI of your AI spending. Here is how to answer with a number you can defend.
I distrust those figures, my own included. At Oliverlist, revenue per person tripled between 2019 and 2023. I can explain that gain: it came from the automation we put in place from 2020, from importing contacts to issuing invoices. It did not come from generative AI. AI joined our product in May 2023 and today sorts prospect replies, writes call scripts and coaches the team. I prefer to look at its effects task by task rather than add them up into one big number.
Measuring the return on AI is not complicated. It mainly takes discipline: a measurement before, few metrics, a measurement after, and honesty about what is still unproven.
Take a baseline first
Without a starting point, there is no return to measure. It is the first step, and the one people often skip, because they are in a hurry to see the tool running.
For two to four weeks, before you change anything, record:
- volume: how many documents, requests or quotes go through the task each week;
- time: how many hours the team spends on it, rework included;
- turnaround: how long it takes between a request arriving and being handled;
- errors: how many corrections, returns or complaints.
Start with what already exists: email timestamps, software logs, ticket dates. For the rest, a simple log kept by the team is enough, as long as you explain why. The point is not to watch people. It is to understand the task.
Write the baseline down, date it and have the people who do the work check it. It becomes your reference. Pick a normal period too: a baseline taken during Ramadan's shorter working hours, the summer slowdown or year-end closing will distort every comparison that follows.
Pick two or three metrics
With ten metrics, something will always look better. Two or three are enough, provided they match what you actually want to change.
| Metric | What it measures | The trap |
|---|---|---|
| Time spent | Hours per week on the task | Forgetting the time spent checking the AI |
| Turnaround | Time from request to response | Looking at the average and ignoring slow cases |
| Errors | Corrections, returns, complaints | Counting only the errors clients notice |
| Volume handled | Files handled per week, same headcount | Comparing periods with different workloads |
Choose according to your problem. If clients wait too long, measure turnaround, not hours. If the team is swamped, measure time and volume. And always keep one quality metric: a faster process that makes more mistakes is not a gain. It is a risk moved somewhere else.
Measure again, with the same rules
The second measurement uses the same method, over a period of the same length and a comparable workload. Not in the first weeks: the team is learning, the instructions are being tuned, and the numbers do not mean anything yet. Wait until usage is stable.
Three costs often disappear from the calculation:
- review time: if every AI output is checked, that time comes off the gain;
- tuning: correcting instructions, handling exceptions, following up errors;
- subscriptions and the tools around them.
Look at the AI's errors as well, not only the team's. A simple method: every week, the person who knows the task reviews a sample of cases handled by the AI and notes what was right, what was wrong and what should have been flagged.
Finally, ask what happens to the time saved. Hours that nobody reuses are a theoretical gain. Decide before launch what the team will do with that time: handle more volume without the next hire, respond faster, cut overtime. That decision is what turns a measured gain into a real return.
Be wary of estimated savings
An estimated saving is nearly always calculated the same way: the time a task takes, multiplied by how often it comes up, multiplied by the share AI would take on. The arithmetic is right. The assumptions rarely are. It ignores exceptions, review time, adoption time, and the fact that nobody spends their whole day on a single task.
I separate three kinds of figures, and I label them as such:
- estimated: calculated beforehand, on assumptions. Useful to decide whether to start a project, never to judge its result;
- measured: observed afterwards, with the same method as the baseline;
- extrapolated: measured over a few weeks, then projected over a year. Use it with care, and say so.
To replace an estimate with a measurement before you commit, run the AI alongside the team for a few weeks, without letting it decide or send anything. Compare its proposals with what the team actually did. You then know how accurate it is on your own files before you hand it anything.
Watch out for mixed effects too: a new hire, a quieter season or a large new client can move the numbers as much as the tool. If two teams do the same work, starting with one before the other gives a fairer comparison.
Some effects take months to show: client satisfaction, retention, perceived quality. Until they are measured, write "not measured yet". It is less flattering than an estimate. It is also what makes the people reading your numbers believe them.
Key takeaways
- No baseline, no measurable return. Take it before you touch anything, over a normal period.
- Two or three metrics among time, turnaround, errors and volume, always including one on quality.
- Measure again with the same rules, after deducting review time, tuning and subscriptions.
- Keep estimated, measured and extrapolated figures apart, and always say which one you are quoting.
- Whatever is not proven yet is reported as "not measured yet".