Why AI pilots stall at 80%, and what the last 20% actually is
The last 20% of an AI pilot is six blockers with six artifacts: a quality bar, an eval report, a cost model, a data-flow map, an escalation design, an owner.
A pilot stalls at 80% because the remaining 20% is not model work and no single person owns it. Six things are usually missing: an agreed quality bar, evidence beyond a demo, a cost model at real volume, answers to the security questions, an escalation path, and a team that will operate it. Each is unblocked by a specific artifact.
The stall has a recognisable shape. The pilot has been working for two quarters. Every meeting ends with someone saying it looks great, and none ends with a launch date. Ask five people what is blocking it and you get five different answers, none of them wrong, none of them a task anyone has been assigned.
Why do AI projects never make it out of pilot
The demo answered one question: can the model do this at all. Everything between that answer and a launch is a decision somebody else has to make, and they need something in writing first. A pilot with no artifacts gives them nothing to sign, so they ask for another demo, which produces the same answer again.
S&P Global Market Intelligence's 2025 Voice of the Enterprise survey, of more than 1,000 IT and business respondents in North America and Europe, found the share of companies abandoning most of their AI initiatives had risen from 17 percent to 42 percent in a year, and that on average 46 percent of proofs of concept were scrapped before production. The figure and its sampling notes are collected in QueryNow's Enterprise AI pilot paradox whitepaper, and that gap is what this page is about.
The six blockers below are the ones that repeat. Each is an open question held by a different person. The pilot stalls when three or four sit open at once and nobody tracks them as one set.
| Blocker | Who is actually blocked | Artifact that unblocks it | Typical size |
|---|---|---|---|
| Nobody owns the quality bar | The launch decision | One page defining good enough, refusal policy and failure classes | 1 day |
| No evidence beyond a demo | Product and support | An eval report over a fixed case set, run on the shippable build | 1 to 2 weeks |
| Security review has open questions | Security and legal | A data-flow diagram with retention and access answers | 3 to 5 days |
| Cost at full volume is unmodelled | Finance and pricing | A cost model from measured per-run cost and forecast volume | 2 to 3 days |
| No path for cases it cannot handle | Support operations | An escalation design with the queue, the payload and the timeout | 3 to 5 days |
| No team has agreed to operate it | Engineering leadership | A named owner, an alert route and a runbook | 2 to 3 days |
The quality bar is a decision, not a measurement
A pilot without a quality bar cannot fail, which is why it also cannot ship. Someone has to write down what counts: the resolution rate the agent must hit on the task it is allowed to do, the answers that are unacceptable at any rate (a wrong dollar figure, an invented policy, a cross-tenant reference), and what the agent does when it is unsure.
Write it as three lines with numbers in them and a name at the bottom. The number need not be defensible on the first pass, only one the owner will stand behind, because every later artifact is measured against it. Teams skip this because it feels premature before the evidence exists, then find the evaluation they built has no threshold to compare against.
The bar also has to name who accepts it. A builder can produce a score. A builder cannot decide that 92% is fine and 88% is not, because that decision trades a support cost against a launch date, and those are the product owner's to trade.
Evidence means an eval report over a fixed case set
The demo is weak evidence, because the person running it chose the inputs. Replace it with a fixed case set drawn from real transcripts and real failures: every bug someone reported during the pilot becomes a case, with the input, the tenant shape that triggered it, and the property that has to hold. Twenty to thirty cases is enough for a first launch and small enough to run on every change.
Score with assertions where the answer is checkable and a judge model where it is not, and keep both the cases and the judge prompt in version control. A suite written from imagination rather than from transcripts stays green while production breaks, which converts an open question into a false answer.
Run it against the exact build you plan to ship, with the model pinned to a dated version rather than a floating alias. A report that scored a prompt from three weeks ago is evidence about a system that no longer exists. The report is one table: case count, pass rate per failure class, the cases that fail and why, and the date and version it ran against.
Security questions close on a data-flow diagram
Security review stalls because the questions are specific and the answers so far have been verbal. Draw the data flow once and answer four things on the page: what leaves your boundary and to which provider, what is retained and for how long, who can read the traces, and what the model is never allowed to see.
The trace store is the part teams miss. A trace that captures raw tool arguments quietly becomes the most sensitive database you own, usually with the loosest access control, because whoever needed to debug something set it up during the pilot. Decide retention and redaction for it before the review. Secrets belong in server-side resolution at call time, never in a system prompt or a tool argument that gets written to a log.
Identity is the other question that comes back. If a tool takes a customer id as an argument the model fills in, the model treats that field as data it may choose, and a prompt injection turns it into a cross-tenant read. Scoping it at the edge of the request instead is the answer security wants to see on the diagram. The engineering side of that work is listed in the demo-to-production checklist.
Cost at volume is arithmetic somebody has to do
The pilot's bill is meaningless because the pilot's traffic is meaningless. Measure cost per run on the real case set, split cached from uncached input tokens, then multiply by the volume forecast the business actually believes. Three numbers change the answer by more than the model choice does.
The first is cache reuse. A stable system prompt read from cache costs a fraction of the same tokens read fresh, so a cost model that assumes today's hit rate is invalidated by a routine prompt edit. State the assumption explicitly.
The second is tool loops. A tool that returns an empty array when its upstream fails reads as a call that did not work, so the model calls again, and the run's cost is bounded by whatever cap you set rather than by the task. The third is retries and escalations, which turn one conversation into several runs.
Finish the model with a per-tenant ceiling and a decision about what a tenant sees when it hits the ceiling. A silent failure at the cap reads as an outage, and support will file it as one. If the feature is priced for customers, the same numbers feed that conversation, covered in build versus buy for an AI platform.
The escalation path needs a destination that exists
Every agent has cases it cannot handle, and the pilot handled them by having the builder in the room. Decide the three exits before launch: hand off to a human queue with the transcript attached, file a structured ticket with whatever fields the model did extract, or refuse plainly and say what the person should do instead.
Name the queue, the payload it receives, the response time the operating team commits to, and the trigger that fires the handoff. For actions that are expensive to undo, put an approval in front of the individual tool call rather than in front of the whole conversation, with a timeout and a defined behaviour when it expires. Support operations is blocked until this is written down, because staffing depends on the volume it implies.
What changes when customers are the users
An internal pilot can launch with four of the six artifacts and survive, because the users are colleagues who will tell you when it breaks. A customer-facing, multi-tenant agent cannot, and three of the blockers change shape.
The quality bar becomes contractual, since a wrong answer given to a customer is a support ticket with a name on it. The security diagram grows a tenancy question with one acceptable answer: identity is resolved at the request edge and enforced below the model, so no prompt can talk the agent into another tenant's data. The cost model becomes per tenant, because one customer's usage can carry the whole bill and a per-tenant ceiling is the only control that stops it mid-run. Wiring an agent into a product that already has customers is covered in adding an AI assistant to an existing product.
The order to produce them
Produce the quality bar first, in a day, because the eval report has nothing to compare against without it and the security and cost artifacts inherit their scope from it. Take the security diagram second, since it is the one most likely to change what you build. Discovering in week six that traces cannot be retained is a rewrite.
Run the eval report third, then the cost model from the same runs, which is the cheapest sequencing available: the case set you built for evidence is the workload you measure cost on. Escalation design comes fourth and can proceed in parallel with everything after the quality bar. Ownership is last and takes two days once the other five exist, because an operating team that says no is declining to inherit a system nobody can describe.
One rule keeps the list honest. If a blocker cannot be written as a question with a named person who would answer it and an artifact that would let them, it is a preference rather than a blocker, and the answer is to narrow the launch until it disappears. The wider set of shipping decisions sits under AI agent deployment.
Where this gets easier
Two of the six artifacts stop being bespoke projects when the thing running the agent already produces them: the eval report comes from suites whose cases can be promoted from recorded executions and scored by a judge with human review of individual scores, and the cost model comes from cost recorded per execution with cached and uncached tokens separated. Runtype records both as a property of running the agent, which leaves the quality bar, the security diagram, the escalation design and the ownership decision, all of them your team's to make.
Frequently asked questions
- Why do AI pilots stall even when the demo works?
- Because a demo answers one question (can the model do this?) and a launch decision needs five more answered by different people: what quality bar counts as good enough, what evidence exists that the bar is met, what it costs at real volume, what happens to customer data, and which team carries the pager. None of those five are model problems, so a plan built from the demo does not contain them. The pilot keeps working and nobody can name a single blocker, because there are six partial ones.
- Who should own getting an AI pilot to launch?
- One named person with the authority to accept the quality bar, usually the product owner for the surface the agent appears in. The common failure is a pilot owned by whoever built it, since a builder can produce evidence but cannot decide that 92% resolution is acceptable and 88% is not. Security, finance and the operating team each sign one artifact; the owner decides when the set is complete.
- How small can the first launch be?
- Small enough that the six artifacts are cheap to produce. One surface, one tenant segment, one task the agent is allowed to do, with everything else handed to a human, cuts the security review, the cost model and the escalation design to a fraction of their full-scope size. A narrow launch that ships produces the production traces a wider launch needs.