← Back to blog
GenAI en productionMay 25, 2026

LLMs in production: five architecture choices that prevent nasty surprises

Structured outputs, retrieval, routing, caching, and evaluations: five practical choices for integrating an LLM into a maintainable product.

An LLM demo may work with a prompt, an API key, and a few lines of code. A product must also handle invalid responses, missing evidence, latency, cost, model changes, and hostile input. That is where architecture begins.

The following five choices make an integration more predictable. They are provider-neutral and can be introduced gradually.

1. Treat model output as untrusted data

When a model feeds an API, request structured output that conforms to a schema and validate it on the server. A well-formed answer can still be wrong: a schema guarantees shape, not truth. Business rules remain in application code.

Create explicit branches for refusals, missing fields, and validation failures. Do not silently turn free text into a payment order, SQL query, or access grant. The OWASP list for LLM applications includes prompt injection and improper output handling among the risks application teams need to address (OWASP GenAI Security Project, 2025 edition).

2. Separate knowledge from reasoning

When an answer depends on internal or recent documents, retrieve a small set of relevant passages first, then ask the model to answer from that evidence. Keep passage identifiers so the interface can display its sources.

Retrieval does not repair a poor knowledge base. Evaluate retrieval — did the system find the right passage? — separately from generation — did the answer follow it? When no document supports an answer, the system must be able to say so.

3. Route according to the actual task

Not every request requires the most expensive model or extended reasoning. A simple router can send stable extraction to a faster model, reserve a more capable model for ambiguous requests, and keep deterministic rules outside the LLM.

Test the router too. Measure quality per category, retry rate, and total cost. A token price alone is misleading when errors trigger repeated calls or human rework.

4. Design context for reuse

Put stable instructions and shared examples at the beginning of the context, followed by request-specific data. This layout enables prefix caching when a provider supports it. OpenAI’s documentation says cache hits require an exact prefix match and recommends placing static content before variable content (Prompt caching, accessed in 2026).

Caching is not a substitute for context design. Remove irrelevant history, summarize intermediate results, and avoid attaching a whole document when one passage will do. That reduces both consumption and noise.

5. Version evaluations with the product

Prompt testing should not stop at a few hand-picked demo examples. Build a set of anonymized real cases or clearly labeled synthetic ones: the normal path, edge cases, refusals, ambiguity, and hostile input. Store an expected result or scoring rubric and replay the set before changing the model, prompt, or data source.

OpenAI’s evaluation guide describes a continuous cycle: define the objective, collect a dataset, choose metrics, compare runs, and evaluate again after changes (Working with evals, accessed in 2026). For subjective criteria, combine automated grading with periodic human review.

What to observe in production

Log the model version, prompt version, selected route, cited documents, latency, usage, and failure reason. Remove secrets and limit data retention. NIST’s generative AI profile provides a framework for connecting these measurements to risk management and governance (NIST AI 600-1, 2024; updated 2026).

Start with one use case and a baseline that does not use an LLM. Keep the system when its effect on quality, turnaround time, or human effort is visible and its risk remains acceptable.

References

Share this article

Read also

LLMs in production: five architecture choices that prevent nasty surprises