Shipping LLM features that survive real users

Building an impressive LLM demo takes an afternoon. Building one that still behaves three months later, under adversarial users, with a cost line that finance will sign off on, is a different discipline entirely. Here is what we put around every AI feature before we let it near production.

An evaluation set before a prompt

The first artefact on any AI project we run is not a prompt. It is a spreadsheet of one hundred or so real inputs with the outputs we would accept, assembled with the people who will actually use the feature. Until that exists, nobody can tell whether a prompt change is an improvement or a regression — including you.

That set then runs in CI on every prompt or model change, scoring exact matches where the task allows it and using a model-graded rubric where it does not. When a change drops the score, the pull request fails. It is unglamorous and it is the difference between engineering and vibes.

Ground the model in your data, and cite it

Retrieval-augmented generation is now standard, but the implementation details decide whether it works. Three things matter more than the vector database you choose:

• Chunk on semantic boundaries, not character counts. Splitting mid-table or mid-clause destroys the meaning you are trying to retrieve.
• Retrieve more than you need, then rerank. A cheap first-pass retrieval of twenty candidates followed by a reranking pass consistently beats a precise retrieval of three.
• Always return citations. Not only for user trust — a cited answer is auditable, so you can tell whether a wrong answer came from bad retrieval or bad generation.

Treat model input as hostile

This is where our security practice earns its keep. Any text that reaches the model from outside your trust boundary — user messages, retrieved documents, scraped pages, email bodies, PDF contents — is untrusted input, and it can carry instructions.

The defences that hold up in our testing:

1. Least privilege on tools. If the model can call a tool, assume an attacker can call it too. Scope every tool to the current user's permissions and never above them.
2. Constrained output schemas. A model that must return structured JSON matching a strict schema has far less room to be steered into arbitrary behaviour.
3. Human confirmation on side effects. Anything that sends, pays, deletes or publishes gets an explicit human approval step. Every time.
4. Separate retrieved content from instructions. Clear delimiting plus an explicit system instruction that retrieved text is data, never commands.

Prompt injection is not solved. Design as though the model will eventually follow a malicious instruction, and make sure that when it does, nothing important happens.

Budget latency and cost per request

Set the numbers before you build, not after finance asks. For each feature we fix a p95 latency target and a maximum cost per request, then design to fit. That usually means routing simple cases to a smaller, faster model and reserving the large one for genuinely hard inputs, caching aggressively on stable prefixes, and streaming responses so perceived latency stays low even when total latency is not.

On our last three deployments this approach cut per-request cost by 60–80% against a naive single-model implementation, with no measurable quality loss on the evaluation set.

Ship with an off switch

Every AI feature we deploy has a feature flag, a fallback path that works without the model, and a dashboard showing volume, latency, cost, refusal rate and user-reported failures. When a model provider has a bad hour — and they do — you want the feature to degrade rather than take the product down with it.

The honest summary

Most of the work in a production AI feature is not the AI. It is the evaluation harness, the retrieval quality, the security boundary and the operational instrumentation. Teams that build those first ship features that last. Teams that start with the prompt ship demos.