Here's the direct answer

Your chatbot invents answers because its knowledge base is thin, out of date, or unreadable — and because nobody gave it permission to say "I don't know." A language model asked a question it has no source for will produce something plausible rather than admit a gap; that's the default behavior, not a fault. Fixing it means fixing what it reads, constraining what it's allowed to claim, and building in escalation to a human. sosSTEVIE builds customer agents on a real source of truth, or tells you not to build one yet.

Key takeaways

Bots don't hallucinate from nowhere. They hallucinate into gaps you left.
"I don't know, let me get someone" is a feature, and almost nobody builds it in.
If your knowledge isn't written down, a chatbot will invent it. A knowledge problem in a technology costume.
A bad chatbot is worse than no chatbot. It makes promises on your behalf, at scale, all night.
Most businesses that want a chatbot need a knowledge base. The bot is the easy part.

The story

A company called me because their chatbot had promised a customer a refund policy that didn’t exist.

Not a vague implication. A specific policy, stated with total confidence, in a friendly professional tone, complete with a timeframe. The customer, entirely reasonably, expected the company to honor it. The company, entirely reasonably, was furious with the robot.

I asked what the bot had been given to read. Long pause. It turned out: the website, a two-year-old FAQ page, and a PDF brochure. Nobody could quite remember who’d loaded them or when.

None of those documents mentioned refunds. At all. Not once.

So when a customer asked about refunds, the bot did the only thing it could. It had no source, and it had been given no way to say “I don’t know.” So it produced the most plausible-sounding refund policy in the world — one assembled from the ten thousand refund policies it had absorbed from every other company on the internet. It was a beautiful refund policy. Someone else’s.

The bot didn’t malfunction. It did exactly what it was built to do, given exactly what it was given. It was asked a question it had no answer to, in a system where “I don’t know” wasn’t an available response.

They wanted me to fix the bot. The bot was the only thing in that story working as designed.

Bots hallucinate into gaps

A document with a hole cut out of it, representing the knowledge gaps where chatbots invent answers

Here’s the mechanism, plainly, because almost nobody explains it and everyone should understand it before they buy one.

A language model is a system for producing plausible continuations. Give it a good source and ask it a question, and it’ll use the source — that’s what retrieval-augmented generation means, and it works well. Ask it something with no source and no instruction to stop, and it’ll still produce a plausible continuation. It has to. That’s the machine.

It doesn’t know it doesn’t know. There’s no internal alarm that fires. The confident correct answer and the confident invented answer are produced by the identical process and arrive wearing identical clothes.

Which leads to the thing nobody wants to hear:

Every gap in your documentation is a place your bot will make something up.

Not might. Will. If a customer can ask it, and you never wrote down the answer, it will produce one. And it’ll be plausible — assembled from every competitor, every industry norm, every standard policy on the internet. Which is precisely why it’ll sound so convincing and be so specifically wrong about you.

This is why chatbot projects fail. People buy a bot to answer questions they never wrote answers to. The bot isn’t the missing piece. The knowledge is.

When you shouldn’t have one

I’ll talk you out of this if it’s the wrong call, so let me do it here and save us both a meeting.

Don’t build one if your knowledge isn’t written down. Fix that first. You’ll get most of the value from the writing-down alone, and if you skip it, you’re building a machine that invents policy on your behalf, unsupervised, at 3am.

Don’t build one if you get four inquiries a week. Answer them yourself. You’ll do it better and they’ll like you more. A chatbot on a low-volume site is a costume.

Don’t build one to avoid talking to customers. People can tell. Being hard to reach is a strategy, and it’s a bad one, and no amount of conversational polish disguises it.

Don’t build one if being wrong is expensive. Regulated advice, medical, legal, financial, anything where a confidently wrong answer creates liability. The bot can route, gather, book. It shouldn’t advise.

Do build one if you’re answering the same twenty questions all day, you’ve written the answers down properly, you’re losing inquiries out of hours, or people need to book, check, or find something at 11pm. That’s genuinely valuable and I’ll build you a good one.

The Honest Agent standard

A handwritten note reading I don't know, the most important feature an AI agent can have

1. Built on a real source of truth. Written down, structured, current. This is the project. Everything else is configuration.

2. It can say “I don’t know.” Explicitly built in, tested, and prioritized over guessing. The single most important thing on this list and the one nearly every off-the-shelf bot lacks, because “I don’t know” tests badly in demos.

3. It knows what it can’t promise. Hard constraints on pricing, policy, commitments, timelines. It doesn’t invent a refund policy because it’s structurally incapable of inventing a refund policy.

4. Escalation that works. A real route to a real human, offered early and often, not buried after six failed attempts. The measure of a good agent isn’t how few conversations reach a person — it’s how quickly the right ones do.

5. It says what it is. No fake names, no invented “customer service representative.” People are fine talking to a bot that’s useful and honest. They’re not fine finding out later that “Sarah” wasn’t real, and that discovery costs more than the bot ever saved.

6. It’s watched. Read the transcripts. Every week at first. The gaps show up immediately — and every gap is either a knowledge base fix or a constraint fix. A bot nobody reads is a bot inventing things nobody knows about.

What it costs, and why there’s a monthly

$1,500 to build. $297 a month to keep it honest.

The setup covers the agent itself: knowledge base construction from your source of truth, constraints and guardrails, the “I don’t know” behavior, escalation routing, testing, and deployment on your site.

The monthly is the part most vendors don’t charge for and should. Here’s what it’s actually doing:

Your knowledge goes stale. You change a price, add a service, drop a product, move an office. If the agent’s knowledge base doesn’t change with it, you now have a machine confidently telling customers things that stopped being true in March. The vectors need rebuilding when the underlying facts move.

The transcripts need reading. Every week for the first month, then regularly. This is where you find the gaps — the questions people are actually asking that your knowledge base has no answer for. Every one is either a documentation fix or a new constraint, and finding them is the single highest-value thing anyone does with a deployed agent.

The models change underneath you. Providers update, deprecate, and re-tune. Behavior that was solid in January can drift. Somebody has to notice and re-test.

A bot nobody maintains becomes a liability slowly, and invisibly. That’s the pattern I have a personal history with. An unmaintained agent doesn’t announce its decline — it just gradually starts being wrong about a business that’s moved on without it.

If you’d rather maintain it yourself, that’s fine and I’ll hand it over documented. But somebody has to do it, and a chatbot nobody owns is the thing that promised a refund policy that didn’t exist.

The bot was working

It invented a refund policy because someone asked it about refunds and nobody had ever written one down. It didn’t fail. It performed exactly to specification, and the specification was the problem.

The bot is the easy part. It always was.

Let’s find out what your bot would actually have to read — $997 Diagnostic →

About Stevie

I build AI agents and I turn down a fair number of the people who ask me for one, usually because what they need is to write their business down first.

I’ve been on the wrong end of a system doing something confidently and invisibly on my behalf. It didn’t look like a failure either. That’s made me particular about building things that admit when they’re stuck.

sosSTEVIE — for when your digital presence is broken, invisible, or about to be. (754) 302-4631

Why Does My AI Chatbot Keep Getting Things Wrong?