YourAIPilotWorks.WhyIsItStillNotLive?

It works when you demo it. It falls over when a real person touches it. And every time you fix one case, another one turns up behind it, so there is no point on the horizon where this obviously ends.

That is the part that wears people down. Not a crash, not a bad model, just the absence of a finish line. Somewhere around month six the questions change tone. Is the technology not ready. Is the team not strong enough. Did we back the wrong thing.

Almost certainly none of those. Here is the short version before anything else. Your pilot answered the question it was built to answer, which was whether a model could do this at all. It said yes. The work that turns that answer into a system real users touch is a different body of engineering, it is largely known, and it is finite. Nobody wrote it down for you, which is why it feels endless.

This page is that list. It is written for the month after the demo went well rather than the week before it.

If your situation is a bit different from this, two neighbours may fit better. Where the problem is that an AI tool generated your application code and it now needs hardening, making a builder-generated app production ready is the closer page. Where you have already decided to build and want the implementation detail on the agent itself, Node.js AI agent frameworks and what ships to production picks up where this leaves off.

Why Do Most AI Pilots Never Reach Production?

Because finishing a pilot does not put you partway through a production system. It puts you at the end of a different piece of work.

You have probably seen the number already. MIT Project NANDA, in The GenAI Divide, State of AI in Business 2025, found that roughly 95 percent of enterprise generative AI pilots produced no measurable effect on profit and loss. The figure came from 52 executive interviews, 153 leader surveys and analysis of 300 public deployments, against enterprise investment they put at somewhere between USD 30 billion and USD 40 billion.

The number gets quoted as evidence that the technology disappoints. Read what they actually concluded and it says something else. MIT put the cause in integration and organisational learning rather than in model capability. The models were not the constraint. The systems around them were never built.

There is something oddly reassuring in that if you are the person whose pilot has stalled. It means the situation is ordinary. It also means the fix is not another round of model evaluation, which is where most stalled projects go next because it is the familiar move.

One caution about that statistic, since it now appears on every page in this space. It measures whether pilots produced measured business return. A pilot that shipped and helped but was never instrumented well enough to prove it counts in the 95 percent too. That is worth knowing, because it points at the same missing habit this whole page is about, which is deciding what you are going to measure before you build.

What Is Actually Different Between a Demo and Production?

A demo is optimised for the input you choose. Production is optimised for the input you do not.

That single sentence explains most of the gap, and it is worth sitting with because the consequences are not obvious. In a demo the person running it picks the question. They know which documents are loaded. If an answer comes out badly they run it again, or they rephrase, and nobody counts that as a failure because the point was to show what the system can do at its best.

None of those three conditions survives a real user. The user picks the question, and it will be phrased in a way nobody on the team would phrase it. The documents are whatever is in the system that morning, including the ones with broken formatting. And a bad answer is not rerun, it is either acted on or it destroys trust.

So the pilot is not a partial production system. It is a complete and correct answer to the feasibility question, which is genuinely valuable and genuinely finished. Treating it as ninety percent of the way to production is where the timeline goes wrong, and it is why the remaining ten percent keeps expanding.

The practical consequence is that the work ahead is mostly not model work. Teams find this difficult, because model work is the interesting part and the part they hired for. What is missing is measurement, boundaries, failure behaviour and observability. Unglamorous, well understood, and the entire difference between a demo and a system.

Why Does It Break the Moment a Real User Touches It?

Four things, roughly in this order.

The input distribution is not the one you tested. The team tested with questions written by people who know how the system works. Real users arrive with typos, with two questions in one sentence, with pronouns referring to something they said earlier, and with requests that are reasonable but outside what anyone considered. This is the most common and the most fixable, because once you have real traffic you can sample it.

Somebody was cleaning the data and nobody logged it as work. This one is quietly responsible for a large share of stalled pilots. Before the demo, an engineer noticed a few malformed records and fixed them by hand. Perfectly sensible, took twenty minutes, never written down. Production is the first time the model meets the raw data, and the drop in quality gets blamed on the model.

Retrieval was never measured on its own. If your system looks things up before answering, and most useful ones do, then a wrong answer has two possible causes. The right document was never fetched, or it was fetched and the model ignored it. Those need completely different fixes. Without a way to tell them apart, teams rewrite prompts for weeks to compensate for a search problem, which does not work and is demoralising to watch.

Latency was invisible at one user. Four seconds while you demo is fine and nobody notices. Four seconds for a support agent handling a queue is an abandoned feature. Worse, concurrency exposes retry behaviour that was never tuned, and a badly set timeout under load can double your traffic against the model at exactly the moment it is slowest.

Notice that none of these is a model defect. That matters, because the instinct when quality drops is to try a different model, and it is the one move that reliably costs a month and changes nothing.

How Do You Know the Model Is Right, Rather Than Feel It?

You need a fixed set of inputs with agreed correct outputs, held still while the system changes. That is an evaluation set, and it is the single most common missing piece in a stalled pilot.

Without it, the unit of progress is opinion. Somebody adjusts a prompt, tries five queries, decides it seems better and ships. A fortnight later something else is worse, and there is no way to attribute it. This is exactly the loop that produces the feeling of no finish line, because you cannot finish something you cannot measure.

The good news is how cheap the fix is. One hundred to two hundred real cases, labelled by somebody who knows the domain, is enough to start. It does not need to be automated in week one. It needs to exist, and it needs to stop moving, because an evaluation set that gets edited whenever it disagrees with you is just opinion wearing a lab coat.

Then split it. Measure retrieval by asking whether the passage containing the answer was in what got fetched, with the model uninvolved. Measure generation by handing the model the correct passage and checking the answer. Two numbers instead of one, and they point at different teams and different fixes.

The change teams describe once this lands is not that quality jumps. It is that arguments end. A proposed change either moves the number or it does not, and the conversation about whether to ship takes minutes instead of a fortnight.

What Has to Exist Around the Model?

This is the inventory. A pilot has almost none of it and correctly did not need it. Every row is a thing that has to exist before real users arrive, and the third column is what you will see if it does not.

Two of those deserve a note because they are the ones people push back on.

Defined wrong-answer behaviour is not a nice-to-have. The model will be wrong, on a knowable fraction of requests, and that fraction is a design input rather than a bug to eliminate. Deciding what the system does at that moment, whether it declines, escalates, shows its source, or asks a clarifying question, is the difference between a system that can be deployed somewhere consequential and one that can only ever be a toy.

The security boundary is not the same conversation as data privacy. If your system reads content that other people can write, then text inside that content can be read as instruction. Anywhere the model can then take an action, that is a path from a document into your systems. The fix is architectural, in constraining what actions exist and what each one can touch, rather than in asking the model nicely. Where this touches tool access specifically, what changes when an MCP server stops being local covers the authorisation side in detail. Where the concern is where your data physically sits and who can see it, the on-premise question is the better read.

What Does It Cost to Run Once Real Traffic Arrives?

This is a question about your infrastructure, so what follows is the set of drivers rather than a figure. Anyone who quotes you a number before reading your system is guessing.

Context size, first and largest. You pay for input on every call and context grows without anybody deciding to grow it. A prompt gains an instruction here, an example there, and six months later every request carries a preamble nobody has read end to end in months.

Retrieval breadth, which multiplies the first one. Fetching twenty passages instead of five feels safer and quintuples the input on every request. This is where the evaluation set pays for itself twice, because it lets you find the point where fetching more stops improving the answer, and that point is usually lower than teams assume.

Model tier, applied per task rather than globally. The reasonable default during a pilot is to use the strongest model everywhere, since you are testing feasibility. In production, classification, extraction and routing frequently run acceptably on something smaller, and the evaluation set is what turns that from a gamble into a measurement.

Retries, which are invisible until they are not. A timeout set for a fast day becomes a traffic multiplier on a slow one, exactly when the system is already struggling.

Caching, the largest unpulled lever. Real users repeat themselves far more than pilot testers do. A meaningful share of production questions are near-identical to one asked earlier that day.

The general rule is that cost becomes controllable the moment it is attributed per request instead of read off a monthly total. Until then it is weather. After that it is a number with an owner.

Is Your Pilot Worth Saving?

Usually, yes, and you are right to be suspicious of anybody who says otherwise before reading it.

There are three honest outcomes and they are not equally likely.

Keep it and build around it. This is the common one. The pilot is a correct answer to the feasibility question and becomes the reference implementation. The prompts carry across. The domain knowledge encoded in them, which took longer to develop than anybody logged, carries across. What gets added is the inventory above. Nothing is thrown away.

Replace one layer. Less common and quite specific. One part is genuinely wrong, most often the data or retrieval layer, because it was assembled in a week to make the demo possible and was never meant to survive. That layer gets replaced and the rest stays.

Start again. Rare, and when it is right the signature is a product one rather than a technical one. The task itself turned out to be the wrong task, or the workflow it was built into does not exist any more. Notice that in this case the pilot still did its job, because finding that out cheaply is what a pilot is for.

If a vendor opens with a rebuild, ask them which specific parts of the pilot they intend to discard and why. Someone who has read it answers in a sentence. Someone quoting a rebuild because rebuilds are simpler to scope, and bill better, will answer in generalities about best practice and modern architecture.

We will say this plainly, because it is the thing that stops people asking anyone at all. The rebuild answer is more profitable for us and it is usually the wrong one. Where your existing team can carry the work once the gaps are named, that is what we will tell you.

Who Should Not Take a Pilot to Production?

Three groups. If you are in one of them, productionising now costs money and buys nothing, and there is a cheaper thing to do first.

Nobody outside the team has used it. If every session has been run by someone who built it, you do not yet know what it is for. Productionising will preserve that uncertainty at much higher cost. Put it in front of five real users in the roughest possible form first. Two weeks of that will change what you build, and occasionally it will tell you not to.

You cannot say what acceptable looks like as a number. Not a feeling, a number. Ninety percent correct on this set of cases, under three seconds, escalating rather than guessing when unsure. Without it there is no definition of done, which means the project cannot end, which is the exact symptom that brought you here. This is a half-day of argument between the people who own the outcome, and it is the highest-value half-day in the whole project.

The real problem is upstream of the model. If the data the system depends on is incomplete, stale, or has no owner, a production AI system on top of it makes that worse rather than better, because it converts a quiet data problem into a visible product one. Fix the pipeline first. The AI work gets shorter and cheaper afterwards, and occasionally it turns out the data fix was the thing that mattered.

A fourth case, worth naming because it is common and nobody says it. If your pilot was itself largely written by an AI coding tool and has never been read by an engineer, the honest first step is not productionising, it is reading it. What AI-generated code tends to get wrong covers what that read should look for.

What Does the Path Actually Look Like?

Boring first, interesting last. The ordering matters more than any individual step, and getting it backwards is the most reliable predictor we have of a project that does not finish.

1. Agree the metric. What does acceptable mean, as a number, decided by the people who own the outcome rather than the people building it.

2. Build the evaluation set. One hundred to two hundred real cases with correct answers. Freeze it.

3. Instrument what already exists. Traces at the request level, so any single interaction can be reconstructed later. Do this before changing anything, because it also gives you the baseline.

4. Set the budgets. Cost and latency per request, attributed rather than aggregated.

5. Define what happens when it is wrong. Decline, escalate, show sources, ask a question. Then build the human path that catches it.

6. Release narrow. Real users, small group, real work. Not a wider pilot, an actual release to a few people who need the thing.

7. Widen on the evidence. The evaluation set and the traces tell you when. Nobody has to have an opinion.

Steps one to five contain no interesting engineering at all, which is precisely why they get skipped and precisely where the risk lives. A plan that opens with model selection or framework choice has started at step six and will discover steps one to five later, under time pressure, with users watching. If the model question is still genuinely open for you, the comparison between the Claude and OpenAI APIs for production use is worth reading, but it belongs after this list rather than before it.

How Do You Know Your Team Can Carry This?

Answer these honestly and you will know without asking anybody.

Can somebody reconstruct exactly what happened in a single user request from last Tuesday. Is there a number that tells you whether last week's change helped. Does anybody own the data the system depends on, by name. If the model returns something confidently wrong at two in the morning, what happens. And has anyone on the team run a system with this shape in production before, as opposed to having built one.

Two or more no answers does not mean you need outside help permanently. It means the first release will take considerably longer than the current estimate and should be planned that way, and that the gap is in production practice rather than in AI knowledge. Those are different hires and different reading.

For what it is worth, the reason we are comfortable being direct about this is that the systems where we have done the work are ones where being wrong was expensive. An exam platform serving more than 10 million requests a minute and an education platform with over 250,000 daily users are not AI systems, but they are systems where the difference between a demo and production is measured in public. The instinct transfers. The specifics are what we would want to read on your pilot before saying anything more definite.