AI hallucination: why "we trained it well" is not an answer

A bot on an online shop started inventing product links.

Not always. Not obviously. A customer asks about a product, the bot answers well — right name, right price, right availability — and adds a link. The link looks perfectly correct. Right domain, product name in the path, shaped like every other link on that site. It simply does not exist. The customer clicks and lands on a 404.

What makes this story worth telling is not that it happened. It is that the correct link was arriving.

Reproducing it showed something uncomfortable

The first assumption was the boring one: the shop probably does not return a link, so the bot fills the gap as best it can. Had that been true, the fix would have been trivial.

It was not true. The data the bot received from the shop contained the correct link, every time. The bot had the right value in hand — and would sometimes write its own instead.

Sometimes. Same question, same data, different answer. Sometimes the real link, sometimes an invented one. That is the most important detail in the whole story, because it breaks the way most people test a bot: you try it three times, it works three times, you conclude it works. A non-deterministic error is not caught that way. It is caught in production, by a customer, at the moment they were ready to buy.

Why a prompt is not the fix

The natural reaction is to fix it with an instruction. You add: “Only ever use the link you were given. Never assemble a URL yourself.” You test it. It works.

And it genuinely will get better. But “better” and “will not happen” are not the same claim.

A prompt is a request, not a constraint. A language model does not execute instructions the way a program executes statements — it treats them as a very strong hint about what text plausibly comes next. A very strong hint is still not a guarantee. If an instruction drops the error rate from one in a hundred to one in a thousand, that sounds solved — until the bot runs for months and a thousand questions becomes a number you hit in a week.

The fix was not a better prompt. The fix was structural, in code: the link a customer sees stopped being something the model composes at all. It is no longer the language model’s decision.

Exactly how that is built is our problem, not yours. What matters to you is a different question — which is the point of this article.

This is not a story about our bot

It would be easy to read this as “their bot was wrong, then they fixed it.” That would be the wrong conclusion, and the wrong lesson.

Every bot built on a language model sometimes invents a fact. Every one. This is not a defect that showed up at our place — it is a property of the technology. The model produces a plausible continuation, and an invented link looks exactly like a real one: same shape, same domain, same structure. The model has no concept of “this exists” as opposed to “this looks the way it ought to.” That distinction is not available to it. So it cannot meaningfully be asked to respect it.

The question is therefore not whether some vendor’s bot hallucinates. It does. The question is what the vendor did about it:

• Nothing — they do not know it happens, because they never looked.
• They wrote an instruction in the prompt — reduced the rate and called that a solution.
• They solved it structurally — the claims that must not be invented cannot be invented, because the model does not compose them in the first place.

All three look identical in a demo. The difference only shows up after six months in production, and usually the customer is the one who sees it.

Why a link specifically is dangerous

It is worth saying why this was a priority rather than cosmetics.

An invented price is a visible error — someone notices and tells you. Invented opening hours too. But an invented link is a silent error: the bot looks maximally helpful, the customer clicks, gets a 404, and leaves. Nobody reports anything. In your analytics it looks like a successful conversation — the bot answered, the customer seemed happy, the conversation ended. The only trace is a lost sale you never connect to the bot.

The worst errors are not the ones you can see. They are the ones that happen at the exact moment someone was ready to pay.

Questions to ask any bot vendor

If you are choosing a bot, these are the questions that separate the serious from the hopeful. Ask them of us too:

1. “How do you prevent the bot from inventing a fact?” If the answer is “we trained it well,” “we use the best model,” or “we tuned its prompt” — it is not solved. Those are three ways of saying “we hope it will not.”
2. “Is that solved in code or in the prompt?” The question is deliberately blunt. The answer is informative either way.
3. “Have you ever reproduced hallucination in your own system?” Whoever has not, does not know whether they fixed it. You cannot fix an error you never watched happen.
4. “Which claims can your bot not invent, even if it ‘wanted’ to?” A good answer is a short, specific list. A bad answer is “well, all of them.”
5. “What does the bot do when it has no data?” If there is no clear answer, the bot is filling the gap — with whatever sounds plausible.
6. “How will I find out that it invented something?” If there is no place where you can see what the bot said and what it could not answer, you will hear it from a customer. Or you will not hear it at all.

The honest claim a vendor can make is not “our bot does not hallucinate.” No such bot exists. The honest claim is: “we know where it can, we know what we closed off in code, and we can show you what happens when it does not know.”

That is the level this work is done at. Everything below it is hope.