foo for thought

If you’ve shipped an AI product in the last year, you’ve probably watched this happen:

1. Demo works.
2. Real users arrive.
3. The model starts confidently inventing nonsense at the exact moment trust matters most.

Most teams blame “hallucinations” like it’s weather. It’s not weather. It’s architecture.

This post is about the practical shift from “LLM wrapper” to production AI system design: retrieval quality, uncertainty handling, evaluation loops, and operational guardrails that keep behavior stable under load.

The Core Mistake: Treating RAG as a Checkbox

A lot of stacks look like this: user query → vector search (top-k=5) → concatenate chunks → prompt model → return answer.

This is not robust retrieval-augmented generation. It’s prompt stuffing with embeddings.

RAG only works when retrieval is treated as a ranking and grounding system, not a pre-processing step.

Build Retrieval Like Search, Not Like Magic

• Hybrid retrieval (dense + BM25)
• Two-stage ranking (retrieve, then rerank)
• Metadata filtering (version, tenant, time)
• Structure-aware chunking (not just fixed token windows)
• Query rewriting to normalize user intent

Uncertainty Is a Product Surface

Users trust systems that can say: “I’m not sure,” “I found conflicting sources,” or “This is based on v2.3 docs.”

• Abstain thresholds
• Conflict detection
• Citation requirements
• Answer-type routing (factual vs creative)

Evaluation: Stop Grading Vibes

“Looks good in testing” is why systems collapse in production. Build standing eval sets with golden facts, adversarial prompts, long-tail phrasing, and regression checks for every retrieval/prompt change.

Track grounded correctness, citation precision, abstain quality, latency, and cost per successful answer.

Agentic Flows Need Contracts

As soon as your system calls tools, free-form text is a liability. Use JSON schemas, pre/postconditions, idempotency keys, retries by error class, and human gates for destructive actions.

Reliability Pattern: Plan → Retrieve → Verify → Answer

A robust pattern for technical assistants:

1. Plan required information
2. Retrieve candidates
3. Verify evidence supports claims
4. Answer only verified claims; abstain on gaps

Final Take

The winners won’t be the teams with the flashiest demo. They’ll be the teams that make truthful behavior the default under real-world mess: stale docs, noisy users, partial context, and latency pressure.

RAG is not a feature. It’s an information reliability pipeline.