The case for fully autonomous AI agents is clear. Give a system an objective and let it work end to end, without waiting for people. In controlled tests, agents reason through complex problems, carry out multi-step tasks and often match or exceed human performance on specific measures. For organisations under pressure to do more with the same teams, the appeal is obvious.
Production is a different environment. Inputs arrive incomplete. Exceptions are common. Regulation, brand and customer trust are at stake in every action. A system that succeeds 90% of the time sounds capable until the remaining 10% shows up in customer conversations, compliance filings, or orders. Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and weak risk controls.
The organisations scaling agentic AI successfully are not chasing maximum autonomy. They design for the right autonomy, place oversight where it adds the most value, and let that autonomy grow as the evidence builds.
Benchmarks are not production
The gap between benchmark and production is not only a model problem that the next release will solve. It is structural. Benchmarks are curated: inputs are clean, the task is defined, and success is measurable. Real operations contain situations no training data captured, cases where the technically correct answer is wrong in context, and moments where judgement, not capability, decides the right action.
Consider a customer asking a financial services system about a fund during a market fall. The technically correct answer explains the fund's holdings. The right answer also recognises an anxious investor, avoids anything that reads as advice, and offers a conversation with an adviser. No benchmark measures that. Production depends on it.
Autonomy is a spectrum, not a switch
Human factors researchers have studied this for decades. Parasuraman, Sheridan and Wickens described automation not as on or off, but as levels: from a system that offers no help, through one that suggests options, to one that acts and informs people afterwards, to one that acts without telling anyone. Different parts of the same task can sit at different levels. A system might gather information fully automatically, suggest a decision for a person to approve, and carry out the approved action on its own.
That model is useful for agentic AI. The question is not "should this be autonomous?" but "which steps should sit at which level, and on what evidence?" A quoting system can assemble documents at a high level of automation, propose prices at a middle level, and leave the final commitment to a person.
The ironies of automation
In 1983, the psychologist Lisanne Bainbridge identified what she called the ironies of automation. The more a system automates, the more the remaining human role consists of the hardest tasks: unusual cases, failures, and judgement calls. At the same time, people who rarely intervene lose the practice they need to intervene well.
Both ironies apply directly to agentic AI. If a system handles the routine cases and passes on only the difficult ones, the people receiving them need more skill, not less. And if they see too few cases, they lose touch with how the work is done. Good design keeps people engaged through regular review, visible reasoning, and a steady flow of real decisions, rather than calling on them only when something goes wrong.
Using AI and governing AI are different roles
Some people use AI as a tool: they apply it to a task and accept its output. Others govern it as a capability: they shape how it works, watch its performance, step in when it drifts and improve how it fits the operation. Using AI captures efficiency. Governing it captures lasting advantage.
Governing involves four responsibilities:
Setting boundaries: what the system will and will not do, which situations need escalation, and which constraints hold whatever the system might otherwise produce.
Watching performance: not only whether the system is running, but whether its outputs remain right, whether behaviour is drifting and whether edge cases are handled well.
Stepping in: recognising when an automated response is not good enough and applying human judgement.
Improving: learning from production and refining how the system and the workflow fit together.
This is skilled work. It needs an understanding of both the system and the operation, and judgement about when to trust the system and when to override it. It is also more interesting work than the routine tasks it replaces, and it is where people's time is most valuable.
Trust should match reliability
Lee and See's review of trust in automation, one of the most cited in the field, makes a simple point with large consequences. The goal is not maximum trust. It is appropriate trust: reliance that matches how reliable the system is in each kind of situation. Too little trust, and people redo the system's work and gain nothing. Too much, and they stop checking where checking matters.
Appropriate trust has to be built on evidence the people involved can see. That means systems that show their reasoning, flag their own uncertainty, and make it easy to compare their outputs with what an experienced person would have done. It also means measuring reliability by case type, not as one overall number, because a system can be excellent on routine work and weak on a rare but important category.
Oversight should change as trust is earned
The right amount of oversight is not fixed. At the start of a deployment, people review every output. That is slow, but it builds the evidence that matters: where the system is reliable, where it is not, and which kinds of cases need a person.
As the evidence builds, oversight can shift from approving each action to setting objectives and limits, monitoring performance, and stepping in when it matters. Drone regulation followed the same path. Pilots first had to keep a drone in sight. As detect-and-avoid systems proved reliable, they could fly beyond it while remaining in command. The move should follow evidence, never a calendar, and binding decisions stay with people throughout.
Monitoring is a daily discipline
Once a system is live, governance becomes routine work. Four signals matter most:
Quality by case type: how often outputs are accepted, corrected or rejected, broken down by the kinds of cases that matter.
Drift: whether behaviour changes as inputs, products, prices or rules change around the system.
Escalation patterns: which cases are passed to people, and whether that mix is shifting.
Overrides: where people disagree with the system, which shows where configuration or training needs attention.
None of these needs sophisticated tooling to start. Each needs an owner and a regular review.
Standards and regulation now expect designed oversight
Governed autonomy is no longer only good practice. NIST's AI Risk Management Framework organises AI governance into four functions: govern, map, measure and manage. ISO/IEC 42001, published in 2023, sets out requirements for an AI management system that organisations can be certified against. Both expect oversight to be designed in, documented and reviewed.
The EU AI Act sets human oversight requirements for high-risk systems, such as those used in credit scoring and recruitment. After the Digital Omnibus, those obligations apply from 2 December 2027. Since 27 July 2026, Article 4 has also required providers and deployers to take measures that support the AI literacy of the people who operate AI on their behalf. Governance designed in from the start meets these expectations without rebuilding later.
People get time back, and better work
Governed autonomy changes what people spend their day on. Routine assembly, data entry and status checks move to the system. Review, exceptions, relationships and improvement stay with people, and grow. The organisation gains capacity without asking anyone to give up the judgement that makes the work good. Done well, the people closest to the work become the ones who shape how the system improves.
Questions to set oversight right
Which decisions bind the organisation, and who must approve each one?
For each step of the workflow, what level of automation fits today, and what would justify moving it?
What would the system need to show, by case type, before review could move from every case to exceptions only?
How will drift be noticed, and who is told?
Who can stop or change the system, and how quickly?
How are the people governing the system kept in practice, trained and supported?
How Mitochondria's agentic AI systems are governed
The approach. Every unit works within boundaries set with the organisation before go-live, with continuous monitoring and a full audit trail. Deployment moves through four stages, and each workflow step sits at the level of automation the evidence supports. Autonomy widens as each stage shows reliability, measured by case type.
What it enables. People review every output at first, then move to setting limits and handling exceptions as the evidence builds. Every step that binds the organisation is attested by someone on its team at every stage. The systems show their working and flag their uncertainty, so trust can match reliability. Overrides and escalations are recorded, so the people governing the system can see where it needs attention.
The impact. Organisations scale agentic AI without taking on risks they have not measured. Their people spend more time on the judgement and relationships only they can provide, and stay in practice on the work that matters most.
Read more about Mitochondria's approach.
Sources
Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics.
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6).
Lee, J. D. and See, K. A. (2004). Trust in automation: designing for appropriate reliance. Human Factors, 46(1).
ISO/IEC 42001:2023, Artificial intelligence management system.
Future of Privacy Forum, AI Act timeline after the AI Omnibus
Mondaq, the Digital Omnibus on AI enters into force (Article 4)
—
Mitochondria builds agentic AI for evidence-led core operations. Research-led and model-agnostic. ISO/IEC 27001:2022 certified, with controls aligned to the GDPR, UK GDPR, India's DPDP Act and Australia's Privacy Act. Amsterdam and Pune. Request a working session →