Why most AI pilots die in production

The most common AI failure mode I see in 2026 isn't the model. It's everything around the model.

A team picks a workflow — invoice triage, customer support deflection, sales-call summarization. They prototype it in two weeks. The output looks good in the demo. The board deck gets a green tick. Six months later the pilot is quietly de-prioritized. The owners shrug. "We learned a lot."

What they learned is that they don't have an AI strategy — they have an AI artefact. The two are not the same thing.

The artefact–system gap

An AI artefact is a prompt, a notebook, a Zapier flow, or a thin SaaS wrapper. It works in narrow conditions. It impresses in demos. It cannot survive contact with production because production isn't a controlled environment — it's a stream of weird inputs, edge cases, drift, and unowned hand-offs.

An AI system is the artefact plus the surrounding infrastructure that lets it survive. The pieces that make a system, and that pilots almost always skip:

• Evals. Not vibes. Concrete, version-controlled tests that fail loudly when the model regresses, when the prompt is changed, or when the upstream data shifts.
• Human-in-the-loop architecture. A clear answer to: which decisions does the model make alone, which does it propose, which must it never touch? And: where does the human exit gracefully?
• Observability. Drift monitoring, error categorization, cost-per-useful-action metrics. Not generic LLM dashboards — instrumentation that maps to your operating metric.
• Ownership. A named human who is paid to operate the system, not just to be cc'd on its outputs. AI without a named owner decays the way unmaintained code does — only faster, because the substrate is also moving.

If you are not building all four alongside the artefact, you are not building a system. You're building a slide deck with side effects.

The honest test

When we audit an AI initiative, the diagnostic question is: if the original engineer left tomorrow, would this still ship the same outcome in three months?

Almost always, the answer is no. And once the answer is no, the rest of the conversation writes itself. Either the team builds the surrounding system, or the artefact is sunset. The middle path — keeping it on life support — costs more than killing it.

What the working model looks like

The teams that ship AI in production reliably tend to have three things in common:

1. They build narrow. One workflow, one model, one well-bounded responsibility per agent. The everything-bot is a graveyard of half-shipped scope.
2. They evaluate before they deploy. Evals are written before the agent is live. The success bar is named in advance, and the team is willing to not ship if the bar isn't cleared.
3. They wire ownership into the org chart. "AI ops" is a real role with real KPIs — usually monthly drift incidents, cost-per-useful-action, and time-to-fix-a-regression. If those numbers don't have a named owner, the system has no immune system.

What it isn't

It isn't bigger models. It isn't fancier vector stores. It isn't agent-of-agents architecture. Those are tooling questions; the failure mode I'm describing is operational. You can build a perfectly functional production AI system with a single LLM call, structured input, and a Postgres row to log every decision. We have. Repeatedly.

The discipline is the moat — not the stack.

So what should you do on Monday

If you have a pilot in motion, run the diagnostic. Sit with the team for an hour and answer four questions, honestly:

• What is the eval suite, and when did it last fail?
• Who is the named human owner, and what fraction of their week is on this?
• What is the cost-per-useful-action, and is it trending?
• What happens if this model is silently down-weighted by the provider next quarter?

If you can't answer all four, you don't have a system. You have an artefact. The pilot is still alive — but only nominally. Decide now whether to invest in the system or shut it down. The thing you can't afford is the third option, which is the one most teams default to: leave it on, hope, lose six months.