The demo went well. The model answered the test questions, the room nodded, and someone said "let's ship it next sprint". Two months later the feature is still behind a flag that nobody turns on.
We see this pattern more than any other in AI work, and it is rarely the model's fault. The model is usually the most finished part. What's missing is the work around it: knowing what good looks like, handling the inputs real users send, keeping data safe, and having one person who owns the result.
This post covers why AI features stall, and the checklist we use to get one into production.
Five reasons AI features stall
1. Nobody wrote down what "good" means
A normal feature has acceptance criteria. An AI feature often has a feeling. "It should answer questions about our documents" isn't something you can test, so nobody can say when it's done. Every review turns into a debate about the one example someone tried that morning.
The fix is boring, and it works: a fixed set of real inputs with the expected output for each, and a pass mark agreed before anyone tunes a prompt.
2. The demo ran on clean inputs
Demos use the five documents the team knows by heart. Production gets scanned PDFs, tables split across pages, emails with three forwarded threads, and questions nobody thought to ask. Quality drops, trust drops with it, and the launch slips while the team chases bad examples one at a time.
3. The risk review comes last
Security, legal and privacy see the feature a week before launch, and ask the questions that should have shaped it. What data goes to the model provider? Who can see the outputs? What happens when it's wrong? Answering them late often means rebuilding part of the feature.
4. The interface asks too much of the user
An empty chat box hands the hardest problem, knowing what to ask, to the user. It also makes the feature hard to test, because the range of possible inputs has no end. A narrower interface is often more useful and easier to make reliable.
On Go Legal AI, documents are created through short guided questionnaires instead of a blank chat. Users answer questions they can answer, and the product does the drafting.
5. Nobody owns it end to end
The data person owns the pipeline, the backend team owns the API, product owns the screen, and the model sits somewhere in between. When quality drops, each part looks fine on its own. An AI feature needs one named engineer who owns the whole path from input to output, and reads every change to it.
The checklist
We work through this list before an AI feature goes to real users. It is short on purpose. Each item is a decision or an artefact, not an aspiration.
- One job, one user. Write the feature as one sentence: who it is for, and the job it does for them. If it takes two sentences, it is two features.
- An eval set before tuning. Collect 30 to 50 real inputs with the expected output for each. Agree the pass mark with whoever signs off the launch. Run the set on every change.
- The ugly inputs, early. Ask your team for the ten worst files, tickets or questions they can find, and add them to the eval set in week one. They tell you more than a hundred clean ones.
- A clear line for what the AI may do alone. List every action the feature can take. Mark the ones that change something outside the system: sending, posting, paying, deleting. Those need a person's approval, at least to start with.
- Personal data kept out of prompts. Decide what the model actually needs to see, and remove the rest before the call, not after. On TalVagt, an AI bookkeeper whose agent system we engineered, personal data is scrubbed before anything reaches the AI.
- Output a person can check. Show the source, the draft or the reasoning, not only the answer. If users can't check it, they won't rely on it.
- A cost and speed budget. Set a ceiling for model cost per request and for response time, and measure the slowest requests as well as the average. Choose models against those numbers, not against a leaderboard.
- Tracing from day one. Log every input, the context the model saw, the output, and what the user did next. Without traces, every bug report is a guess.
- A flag and a small first group. Ship to a handful of real users behind a feature flag. Read their traces every week. Widen the group when the numbers hold.
- One owner. One senior engineer accountable for the feature, who reads every change to it, including the ones a coding agent wrote.
None of this is new. It is the same discipline you would apply to payments or sign-in. AI features just make it easier to skip, because the demo looks finished long before the work is.
Scope it in stages
The other habit that keeps AI work moving is staging. Don't plan one big launch. Plan a path where each stage proves one specific thing, and has a test for moving on.
Sengira, a privacy-first home companion that helps older adults live on their own for longer, went from proof of concept to MVP Light to MVP Full. It isn't a chatbot, but the lesson carries over. Its hard part was the device layer: consumer sensors don't expose every signal the same way, so the integration had to be planned sensor by sensor. That kind of problem only shows up when you test on real hardware, or real data, early. The platform also has 51 safeguards, and each one was stress-tested. That is the unglamorous work that makes alerts worth trusting.
For an AI feature, the stages might look like this:
- Proof of concept, one to two weeks. Does the model handle your hardest real inputs at all? You leave with a score on the ugly inputs and a cost per request.
- Internal beta. Your own team uses it on real work. You leave with traces, a list of the ways it fails, and fixes for the top three.
- Small external group. Behind a flag, with the approval steps in place. You leave knowing whether people come back and use it without being asked.
- General release. Only when the eval pass mark and the cost budget both hold.
If a stage fails its test, you have learned something cheaply, instead of at launch.
What to do this week
If you have an AI feature that has been "nearly ready" for a while, three questions will tell you where it is stuck:
- Is there an eval set? If not, building one is the first job, before any more prompt changes.
- Who owns the feature end to end? If the answer is a team, name a person.
- What do the traces say? Read 20 real ones yourself. Within an hour you'll know whether the problem is the inputs, the retrieval, the prompt or the interface.
This is the work our engineers do when they join a team. They plan first, agree what done means with you on day one, and use coding agents for the volume, while a senior engineer reads every line before your team sees it. How we work has the details, including the free first week.