In February 2024, a Canadian tribunal ordered Air Canada to honor a bereavement fare policy that did not exist. The airline's website chatbot had invented it, and a grieving customer had planned around it. The case is Moffatt v. Air Canada, and it is the cleanest cautionary tale about AI chatbot hallucinations we have, because of one detail: nothing malfunctioned. The bot did what language models do when the real answer is out of reach. It produced a plausible one.
If you are putting a bot in front of customers, that detail should reorganize your priorities. Hallucination is the default behavior of an unguarded language model. The job is to build the conditions where the default cannot operate, and those conditions are specific, testable, and mostly unglamorous.
Why language models make things up
Three plain-English reasons. No math required.
A model is trained to continue text plausibly. The most useful mental model I know is autocomplete with very good manners. Ask it a question and it produces the shape of a good answer: confident structure, appropriate level of detail, specifics right where readers expect specifics. When the truth is available to it, plausible and true mostly coincide. When the truth is missing, plausible means "sounds like answers I have seen". A bereavement policy that resembles other airlines' policies is exactly what you would expect the machine to produce. Producing it is the machine working as designed.
"I don't know" is rare in the training data. The internet the model learned from is mostly confident text: documentation, articles, forum posts by people who showed up specifically to answer. People rarely publish pages announcing that they are not sure. So the sentence a support bot needs most is the one the model has practiced least.
Confidence is a style, not a signal. The model writes with the same fluency when it is right and when it is wrong, because tone is learned from text rather than derived from certainty. Humans calibrate trust on tone. That mismatch is the whole trap, and it is why a hallucination reads as more convincing than an honest hedge. Try it with something that cannot have an answer: ask a plain model about a product you invented this morning, and enjoy the specifications.
Why support is the worst place for it
A general-purpose chatbot inventing a fact produces a screenshot and some laughing. A support bot inventing a fact produces a commitment.
Your bot speaks from your domain, next to your logo, in your brand voice. To the customer there is no meaningful difference between "the bot said" and "the company said". The tribunal saw it the same way. A wrong answer about your own refund policy is an offer you did not mean to make, priced at whatever the customer heard.
Scale makes it worse. A human agent who does not know the answer improvises once, to one person, out loud. A bot improvises in writing, at 3am, to everyone who asks, and every transcript is a record. Support answers also get acted on. People book travel, ship packages back, and make purchases around what your bot tells them, which is how an invented policy turns into a tribunal date.
There is also the slower cost. Catch your bot inventing something once and every future answer becomes suspect. Trust in a support channel fails all at once, and rebuilding it takes far longer than the setup work that would have protected it.
What actually prevents it
There is no single mechanism. There is a stack, and this is roughly the order I would add it in.
Ground every answer in retrieved content
Retrieval-augmented generation: before answering, the system fetches the relevant parts of your indexed content and instructs the model to answer only from them. The bot's raw material changes from "the internet, remembered" to "your pages, quoted". I wrote a plain-English explainer on RAG if you want the mechanics. Grounding narrows the space where invention can happen. On its own it does not close that space, which is why the rest of the stack exists. It also changes what improvement means: a bad answer stops being a mystery inside a model and becomes a content problem you can fix by editing a page.
Refuse below a similarity bar
Retrieval always returns the nearest content, and nearest is sometimes nowhere close. Good systems score how well the best matches fit the question, and below a threshold they decline to answer: I do not have that information, would you like a person? This is the single highest-value behavior in the stack. An unanswered question is recoverable. A confidently answered wrong one is a policy you now allegedly have. Tuning the bar is a product decision you will feel: strict declines more borderline questions, loose stretches further. Strict is the right default for anything involving money.
Cite sources on every answer
Citations make answers auditable. When the bot cites the returns page, anyone can check the returns page. They also change how you operate: skimming transcripts, you can spot the weak or outdated source behind a bad answer and fix the source, which fixes every future answer drawn from it. If a tool cannot show where an answer came from, you cannot audit it, and neither can the customer arguing with it. Citations also discipline the writing itself: an answer that has to point at sources is an answer that stays near them.
Never let the bot compose a URL
Links are the easiest thing in the world to hallucinate, because URLs follow guessable patterns and a model completes patterns for a living. yoursite.com/returns-policy looks right whether or not it exists. The rule that works is mechanical: the bot never writes a URL from scratch; it only re-shares links that already exist in your indexed content. We enforce that in Hey Support at the system level, because asking nicely does not survive contact with a language model.
Set topic boundaries
Decide what the bot will not discuss: medical or legal advice, competitor comparisons, whatever is radioactive in your business. Then enforce it as scope rules in the system, where it is checked on every message, instead of a paragraph of hopes. A smaller surface has fewer places to be wrong in. The list is usually short and obvious once you actually write it down. Writing it down is the step teams skip.
Keep a person one click away
Handoff is the pressure valve for everything above. Without it, refusals turn into dead ends, and frustrated visitors start trying to force an answer out of the bot, which is exactly the pressure under which guessing gets rewarded. With handoff, "I don't know" becomes a route: the bot collects context, a human picks up the thread. The failure mode of a well-built bot is a conversation with a person. That was the point all along. Async counts, too. A small team that replies within the hour, with the full transcript in front of them, beats a bot cornered into guessing at midnight.
What does not work
Begging in the prompt. "Do not hallucinate" and its longer cousins shape style, not knowledge. You cannot instruct facts into existence. The model hedges a little more gracefully and invents the same policy.
Temperature zero. Temperature controls how random the wording is; it says nothing about truth. At zero the model picks the most probable continuation every time, and when the truth is out of reach, the most probable continuation is still fiction. You get deterministic wrongness, which arguably reads as more trustworthy because it is so consistent.
A newer model. Each generation calibrates better, and each generation still guesses when the answer is missing from what it was given, because the missing answer is the problem. Model quality moves the frequency. Grounding moves the category.
Longer disclaimers. "AI may produce inaccurate information" in small gray text changes nothing about the answer above it. It footnotes the wrong answer instead of preventing it.
How to test yours before customers do
The useful test is adversarial. Sit down for an hour and try to lose.
- Ask questions your content does not answer. This is the big one. The bot should refuse and offer a path forward. If it answers anyway, you have found the guessing, and everything it said in that answer came from nowhere.
- Ask for links. "Where can I read the warranty policy?" Then click every URL it gives you.
- Ask about competitors. It is bait for comparisons and opinions you never wrote and would never sign off on.
- Push for promises. "So you guarantee delivery by Friday?" "Can I return this after 90 days if it is unopened?" Watch whether pressure produces policy.
- Rephrase one real question five different ways and check that the answers agree with each other.
Run the hour before launch, then again whenever your content changes shape. Your first week of real transcripts will find gaps the hour did not; customers are better at asking unanswerable questions than you will ever be. What the bot can know is decided upstream, when you index your content; how to train an AI chatbot on your website covers that side, and the operational rules around fallbacks and handoff are in our chatbot best practices list.
The principle underneath all of it: honest over impressive. A bot that says "I don't know, let me get someone" reads as modest in a demo, and completely different in month three, when the transcript archive contains zero invented policies. An honest refusal that routes to a human is a feature. A confident guess is an incident report you have not read yet. Every mechanism above ships in Hey Support; if you want the guessing designed out from the start, try it free and spend your first hour trying to break it.
