01 / An $812 Hallucination
In November 2022, a man named Jake Moffatt asked Air Canada's website chatbot about bereavement fares. His grandmother had just died. The bot told him he could book a full-fare ticket and apply for the bereavement discount within ninety days of travel.
That policy did not exist. Air Canada's actual bereavement rules, published on a different page of the same website, prohibited retroactive applications. Moffatt booked the full fare anyway, applied for the refund, was denied, and sued.
Fifteen months later, a Canadian tribunal ruled in his favour. Air Canada owed him $812. More importantly, the airline's defense that the chatbot was "a separate legal entity responsible for its own actions" was rejected. The tribunal ruled the company was bound by what its bot said, the same way it would be bound by what its website said. The chatbot was on the website.
Their bot didn't hallucinate from a malicious prompt. It cited a policy that contradicted another page on the same website, and nobody knew.
That last part is what should keep founders up at night. Not the lawsuit. The detection gap. Air Canada had a chatbot in production, talking to paying customers, telling them things that contradicted the company's actual policies, and the company had no way of knowing until somebody sued them.
This is the modal failure mode of RAG in 2026. Not malicious. Not exotic. Just a confident, articulate answer that quietly disagrees with the truth, in production, undetected, until a customer notices.
02 / The 60/40 Frame
Every RAG vendor sells you the same 60%
Vector DBs. Embedding APIs. Frameworks. Re-rankers. The whole RAG ecosystem is built around five steps that fit in an afternoon's integration work and a clean demo. Pinecone, Weaviate, Qdrant. OpenAI embeddings or Cohere's. LangChain or LlamaIndex. Cohere Rerank or a cross-encoder. Pick four, wire them up, ship a quickstart.
That stack is the visible 60% of RAG. It's well-documented because it's well-bounded; each step has a clean input, a clean output, and a benchmark you can run before lunch.
The other 40% is the part nobody sells you. It's not a step in the pipeline. It's the connective tissue that decides what your system does when reality stops cooperating.
If your RAG breaks in production, the cause is almost never in the part of the stack that a vendor will sell you.
The 60% gets the marketing budget because that's where the ecosystem competes. None of those vendors wants to own the part that requires you to understand your own users, your own documents, your own edge cases. So thousands of guides got written for the easy half. The hard half stayed quiet, the private knowledge of teams who already shipped, broke something, and lived through it.
It's four systems. Most teams skip them or build them by accident.
03 / The Four Invisible Systems
Eval harnesses
"It feels better" is the most expensive sentence in modern AI engineering. It's what teams say right before a regression that costs them retention for two months. Air Canada's chatbot probably "felt fine" for the entire eighteen months it was misleading customers about bereavement fares.
A real eval harness has three pieces. A golden set of fifty-plus question-answer pairs pulled from your own production logs, labeled by humans who know the domain. Automated scoring across faithfulness, answer relevance, context precision, and context recall. Ragas and TruLens make this tractable, LangSmith and Arize Phoenix wrap it for production. And a loop where every change runs against the harness before shipping, with versioned scores you can compare.
Most teams don't skip this for technical reasons. They skip it because labeling 200 question-answer pairs is the least demoable work in the entire stack. Nobody wants to present that at the all-hands. The teams that build it anyway are the ones that survive procurement.
Freshness pipelines
Ask a team how they handle document freshness, and you'll hear the same answer: we re-index nightly. That solves one of four problems.
Source drift is when your documents change, prices move, policies update, or products ship. Re-indexing solves it, assuming you actually know when sources change. Embedding drift is when you upgrade your embedder; the day you switch from text-embedding-ada-002 to text-embedding-3-small, your old vectors are subtly incompatible with new query vectors. Most teams find out by accident. Relevance drift is when the world changes around your index; new terminology enters the domain, users start phrasing things differently, the retriever keeps doing what it was tuned to do, but it's no longer what users need. Expectation drift is the worst one. Users learn what your system is good at, stop asking the questions where it fails, and your metrics improve while your product silently shrinks.
Each one has a different signal and a different fix. A cron job is not a strategy.
Fallback policies
What does your system do when the retrieval comes back empty? In most production RAG, the answer is: it hallucinates confidently. The retriever returns marginal chunks. The generator, told to answer based on the context, produces a fluent paragraph that drifts further from the truth as the context drifts further from the question. The user gets a wrong answer with citations. They believe it because it's articulate.
A confident wrong answer is worse than a clear refusal. Air Canada's tribunal bill is what it costs at the low end.
A fallback policy is the explicit set of rules for what your system does at the boundary of what it knows. A confidence threshold below which it declines to answer. Routing for out-of-scope questions to search or a human. Visible uncertainty in the UI when the system is guessing. Predictable degradation when one stage of the pipeline fails.
Most teams have none of this. They have a default prompt that says "answer based on the context" and hope nothing weird happens. The hope is doing all the work.
Query rewriting
This is the contrarian one. Spend an hour in your production logs, and the pattern is impossible to miss. A third to half of user queries are malformed for retrieval. Pronouns without antecedents. Multi-hop questions the retriever has to answer in one shot. Questions phrased in ways no document would ever match.
A user types "what about the new one?" The retriever has no idea what "the new one" refers to. It can't read the previous turn. The user isn't wrong to talk that way; humans talk that way. The system is wrong to take the question literally.
Query rewriting fixes this before retrieval runs. On the floor, it's a small LLM call that takes the chat history and the latest message and produces a self-contained search query. At the ceiling, it does decomposition for multi-hop questions, expansion for recall, and intent classification for routing to the right index or tool. Most teams haven't built the floor.
The teams that figure this out late always discover it the same way. Six weeks of tuning chunk size, quality moves plus or minus 2%, an engineer adds a query rewriter as a side experiment, quality jumps double digits in a day. Six weeks of effort on the wrong layer.
04 / What Air Canada Actually Got Wrong
Three failures, one bot
Air Canada's chatbot wasn't one broken thing. Multiple invisible systems were missing simultaneously.
It had no eval harness sensitive enough to catch the contradiction between the bot's output and the company's actual published policy on a different page. That's a faithfulness failure that would have shown up on day one of an honest eval loop.
It had no freshness pipeline that could detect that the bot was citing a policy version that didn't match the canonical source. The canonical page existed on the same website. Nobody checked the bot against it.
And it had no fallback policy. Faced with a question about retroactive bereavement claims, a question its training and indexing didn't cover correctly, it didn't refuse, didn't hedge, didn't flag uncertainty, and didn't route to a human. It made up a policy and stated it as fact.
$812 is what that cost when one customer noticed. Whatever the multiplier was for the customers who didn't notice, didn't sue, and quietly churned, nobody knows. That's the real number, and it's unknowable by design.
05 / Self-Diagnostic
Score your own system
One point per statement that's unambiguously true today, not aspirationally, not partially.
Versioned eval set with at least 50 production-derived question-answer pairs that runs against every change before it ships.
Faithfulness and context relevance are measured separately, not just retrieval recall.
Embedding drift, source drift, and query-pattern drift are detected as three separate problems.
Explicit refusal path: the system declines to answer below a confidence threshold, and low-confidence answers are visibly marked in the UI.
Queries are rewritten before retrieval, especially in multi-turn conversations.
Pipeline logging at every stage; any user session is replayable, and a new engineer could understand the system without reading Slack history.
What your score means
5-6 Real RAG product. Will survive enterprise procurement and a bad week of usage.
3-4 Mid-build. Visible stack works in the demo. Two or three invisible systems are missing where retention quietly leaks.
0-2 Demo, not product. Fine for a pilot. Won't survive a serious customer or a security review.
06 / What to Build Toward
The 60% builds the demo. The 40% builds the business.
Demos win meetings. Trust wins renewals. The visible stack is the price of entry. The invisible 40% is what stops your customers from comparison-shopping the day a competitor demos something that looks identical to yours.
Air Canada paid $812 for one missed answer. The teams pulling away from the pack are the ones who decided that paying $812 or $8,120, or $812,000 wasn't the kind of risk you take while you keep tuning chunk size. They built the 40%. They stopped competing on the part of the stack that anyone can buy.
The visible 60% closes the first deal. The invisible 40% closes the second one.
If three or more of those statements are missed.
A two-week engagement focused entirely on the invisible 40%. We map the pipeline, instrument the gaps, and deliver a written report of what's broken, what's fragile, and what to build next in priority order, with effort estimates.
Diagnostic of eval coverage, freshness, fallback behavior, and query layer graded against the framework above.
Replayable trace of fifty production queries with annotated failure modes and the fix for each.
Ninety-day build plan, sequenced, with the smallest set of changes that move quality the most.
The work itself is scoped tightly. Book a call if your scorecard left you uncomfortable.
Book a Discovery Call
