A peculiar problem appears when AI deployments succeed: the value becomes invisible. Before the system, the pain was tangible. Delays were felt. Errors were noticed. Bottlenecks were complained about. People remembered waiting, deals lost to slow replies and knowledge that left with a colleague.
After a few months of reliable operation, that memory fades. Quotes go out. Enquiries get answers. Information moves where it should. Smooth operation is, by nature, unremarkable.
This makes evaluation hard. The comparison people reach for- what this costs against what we paid before- often misses the point. And when the old process was never measured, which is common, the comparison is harder still.
The wrong comparison
The most common mistake is comparing the cost of a system with the cost of hiring someone to do the same work. It fails because a person cannot do the same work. Nobody can guarantee replies within a minute at any hour, absorb volume spikes without strain, ensure every output carries every required element, or keep a complete, searchable record of every interaction.
That is not a criticism of people. It describes human work: variability, availability and the ordinary limits of attention and memory. A well-designed system solves a different problem, freeing the people around it for the work that needs them.
The better question is what it would cost to build the same capability any other way. Often, the answer is that you can't build it by hiring at any reasonable cost.
The baseline problem
Evaluation needs a baseline, and many organisations lack one. They know things were slower but did not track how much. They know they missed leads but didn’t count them. This is not negligence. Struggling processes rarely get measured: an overwhelmed team has no time to build dashboards about being overwhelmed. The missing baseline is itself evidence of strain.
Douglas Hubbard's work on measurement makes a useful point here: anything that matters can be measured well enough to make a decision, if you are willing to accept a calibrated estimate rather than perfect precision. The question is not "what exactly did the old process cost?" but "what range is it reasonable to believe?"
Reconstructing the baseline
Activity-based costing, developed by Robert Kaplan and Robin Cooper, offers a practical method. Instead of looking for a number in historical reports, trace the activities the work required: the steps, the people, the time each took, the handovers and what happened when someone was away. Then estimate what that process produced under realistic assumptions about human performance.
How long does each step take by hand? How often is the right person unavailable? How often do errors occur at each handover, and what do they cost, directly and downstream? The reconstruction does not need to be precise. Even conservative estimates usually reveal a large gap between what the process could produce and what it did under real constraints.
Five dimensions of value
An evaluation that counts only cost savings misses most of what changed. Five dimensions capture it better.
Capacity returned. The hours of routine work the system now performs, and what the team does with that time instead. The value comes from redeployment: the same people doing more valuable work.
Opportunity recovered. What skilled people were not doing while they handled routine work. A senior salesperson who spent two hours a day assembling quotes now spends them with customers. This rarely appears in accounts, but it compounds.
Errors and risk removed. Manual processes produce manual errors. On high-value transactions, one error can cost a deal and the trust behind the next. Systems that enforce completeness remove whole categories of error. Quality management has long called this the cost of poor quality: the hidden cost of rework, complaints and lost business.
Speed and responsiveness. Where several suppliers receive the same enquiry, speed affects who wins. The advantage is invisible until you lose it.
Scale without strain. Work handled by people grows roughly in line with headcount. Work handled by a system can grow without the same strain, so growth no longer depends on stretching the team.
The assets a deployment leaves behind
A system needs structured product data, documented pricing logic, formal process flows and clear escalation paths. Building them is often the first time an organisation has put its working knowledge into a usable form. That knowledge, once fragile and held by a few people, becomes infrastructure: it reduces dependence on individuals, speeds up onboarding and makes consistency possible. It would have cost a great deal as a standalone project. It belongs to the organisation.
The visibility dividend
Systems that run operational workflows record them as a by-product. Every transaction is logged. Patterns appear: which products draw the most enquiries, when demand peaks, which customers need the most handling, which steps create the most friction. That visibility informs product, staffing and process decisions, and lets management work from evidence rather than instinct. It has value independent of the automation itself.
The counterfactual test
Economists evaluate a policy by asking what would have happened without it. Donald Rubin formalised this as the potential outcomes framework: the effect of an intervention is the difference between what happened and what would have happened otherwise. The same question clarifies AI value: what happens if the system is switched off tomorrow?
The answer usually shows the system has become infrastructure, not a tool. Switching it off does not mean returning to the old process. It means operating without capabilities the old process never had: instant replies, consistency, complete records and room to grow. The organisation is not paying for a small improvement on its past. It is paying for capabilities with no easy substitute.
Value that compounds
A system that has run for a month has little experience to draw on. One that has run for a year has built up patterns and precedents that make it more capable. One that has run for three years holds advantages that a new implementation would take years to rebuild. Early evaluations capture operational improvement. Later ones reveal capabilities that didn't exist at deployment and couldn’t have been bought. The question is not only "what is the value now?" but "what is the trajectory?"
Six questions for evaluating an AI deployment
What capacity has been returned, and how is it being used?
Which errors and risks have been removed, and what did they cost?
What speed and responsiveness have been gained, and how do they affect wins and customer satisfaction?
What lasting assets has the deployment created?
What visibility exists now that did not before?
What would be lost if the system were switched off?
The answers rarely produce one number. They produce a picture of how operations have changed and what is now possible, which is what decision-makers need.
How Mitochondria's agentic AI systems are evaluated
The approach. Every deployment begins by mapping the process as it runs, which also reconstructs the baseline: the steps, people, handovers and failure points. Outcome measures and leading indicators are agreed before go-live, and each unit records what is needed to track them.
What it enables. Evaluation across all five dimensions, with a counterfactual view of what the system provides. Decision memory and complete records show how value grows over time, not only at launch.
The impact. A clear, honest picture of what a deployment has changed, which tells the organisation where the next investment will compound.
Read more about Mitochondria's approach.
Sources
Hubbard, D. W. (2014). How to Measure Anything (3rd edition).
Kaplan, R. S. and Cooper, R. (1998). Cost and Effect: Using Integrated Cost Systems to Drive Profitability and Performance.
Juran, J. M. (1951). Quality Control Handbook (on the cost of poor quality).
Rubin, D. B. (1974). Estimating causal effects of treatments in randomised and nonrandomized studies. Journal of Educational Psychology, 66(5).
—
Mitochondria builds agentic AI for evidence-led core operations. Research-led and model-agnostic. ISO/IEC 27001:2022 certified, with controls aligned to the GDPR, UK GDPR, India's DPDP Act and Australia's Privacy Act. Amsterdam and Pune. Request a working session →