1. What Makes AI Different From Conventional Software
Artificial intelligence is the field concerned with getting machines to do things that have usually required human intelligence. Recognising a pattern. Reading a document. Making a judgement under ambiguity. The definition is broad on purpose, and it has been broad since the 1950s, which is part of why the word carries so much confusion now.
Most of that confusion clears once you understand a single difference.
Conventional software follows rules that a person wrote. Someone sat down, worked out the logic, and encoded it. If the tax rate changes, an engineer changes the rule. The system does exactly what it was told, every time, and when it encounters something the rules did not anticipate, it either fails visibly or does something obviously wrong.
An AI system learns patterns from data instead of following rules that were written out. Nobody encoded the logic by which it recognises a defective weld or drafts a summary. It inferred that logic from examples, and it applies what it inferred to inputs it has never seen. The behaviour was learned, and nobody can point to the line of code that produced any particular output.
That is the whole difference, and everything else in this series follows from it.
Why this matters more than it sounds
Conventional software fails predictably. Give it the same broken input twice and it breaks the same way twice. You can write a test for every case you care about, run the tests, and know where you stand. Quality assurance for conventional software is essentially the discipline of enumerating what could go wrong.
Learning systems do not offer you that. They fail on inputs nobody anticipated, in ways nobody predicted, and often without announcing that anything has gone wrong at all. A document-processing system that has handled four thousand specifications correctly may misread the four thousand and first, and it will not flag its own confusion. It will produce a plausible answer.
This is not a defect that better engineering will one day eliminate. It is a property of how these systems work. They generalise from patterns, and generalisation means behaving on unfamiliar inputs in ways that were never explicitly decided.
What it changes for you
Three practical consequences, and they are the reason this distinction is worth a whole article.
Testing changes. You cannot enumerate the cases, so acceptance testing shifts from proving correctness to establishing a rate. How often is it right, on inputs that look like our real inputs, and what happens on the occasions it is wrong. A vendor who offers you a clean pass or fail on an AI system either does not understand what they have built or is hoping you do not ask.
Sign-off changes. With conventional software, the residual risk after testing is small and technical. With a learning system, there is always residual risk, and somebody in your organisation has to own it. That person should be named before deployment, and should understand what they are owning. In our experience this conversation, held early, prevents a great deal of unpleasantness later.
The meaning of working changes. A system that is right ninety-six per cent of the time is either excellent or unacceptable, and which one depends entirely on what happens in the other four per cent. If a wrong answer produces a draft that a human reviews, ninety-six per cent is very good. If a wrong answer sends an instruction to a factory floor, ninety-six per cent is a serious problem. The number is meaningless without the consequence attached to it.
This is why we have written elsewhere that in agentic systems, the question stops being what the software can do and becomes how it behaves. The unpredictability is inherent. What is designable is the structure around it: what the system is allowed to do unsupervised, where a person stands in the path, and what happens when the system is uncertain.
What we are still working out
We do not know how much of the unpredictability is fundamental and how much is immature engineering. Our working assumption is that a good deal of it can be contained by design, through mandate boundaries, escalation paths, and human review at the points that matter, and our deployments are built on that assumption. But we hold it as a bet, not a certainty. It is possible that these systems will remain less legible than we would like for longer than the industry expects, and if that turns out to be true, the case for keeping humans in the path gets stronger rather than weaker.
We are also unsure how the acceptance-testing question resolves at scale. Establishing an error rate on representative inputs is straightforward for one workflow. Doing it across twenty workflows in an organisation, continuously, as models change underneath you, is a problem nobody has solved elegantly. We are working on it, and we do not think anyone has the answer yet.
—
Mitochondria is an agentic AI product company based in Amsterdam and Pune. ISO 27001:2022 certified. Classified within the limited and minimal risk tiers of the EU AI Act, with controls aligned to the GDPR, UK GDPR and India's DPDP Act.