What a website chatbot can and cannot do

"AI chatbot" covers two genuinely different things, and confusing them is where most disappointment comes from. One is a general-purpose model answering from everything it was trained on, with no particular knowledge of your business beyond what is in its training data. The other is a chatbot grounded in your own content — your pages, your documents, your policies — that answers only from what you actually gave it. A website chatbot worth having is the second kind, and understanding the difference is most of what you need to judge whether one is right for your business.

Grounded answers versus a general model

A grounded chatbot works by searching your own content for the passages most relevant to a question, then generating an answer from those passages rather than from general knowledge. That means it can quote your actual returns policy instead of guessing a plausible-sounding one, and it means an answer can be traced back to the document it came from. A general model, asked the same question with no grounding, will produce something fluent and confident whether or not it is correct for your specific business — it has no way of knowing what your policy actually says.

This is worth checking before you buy, because "AI chatbot" is used loosely enough in marketing that the distinction is often not obvious from a demo. Ask directly what the assistant is answering from, and ask what happens when it does not know — a supplier who cannot give a clear answer to either question is probably selling the general-model kind, dressed up in your logo and colours.

Why declining to answer is the important feature

The engineering that matters most in a chatbot like this is not how naturally it writes, it is what it does when your content does not cover a question. A well-built assistant is deliberately trained to recognise when retrieval has not returned anything relevant and to say so, rather than filling the gap with something invented. A chatbot that always sounds certain, on every topic, has not solved the hard problem — it has hidden it. Before launch, this behaviour should be tested on purpose, with questions your content genuinely cannot answer, to confirm the assistant declines rather than improvises.

When it should hand over to a person

Declining is only useful if there is somewhere to go next. A grounded assistant should recognise the point at which a question needs a human — because it is outside its content, because it is a complaint, because the visitor asks for a person directly — and hand over cleanly, ideally with the conversation attached so nobody has to repeat themselves. A chatbot with no handover path just leaves a frustrated visitor at a dead end.

What to measure once it is live

The most useful thing a chatbot produces after launch is not the conversations it handles well, it is the log of questions it could not answer. That gap report tells you what visitors actually want to know that your existing content does not cover — often a more direct signal than analytics on the pages themselves, because it is visitors asking in their own words rather than clicking through a menu you designed. A chatbot that never fails is not necessarily doing well; it may just be that nobody is asking it anything difficult, which is worth checking against how many conversations it is actually having.

Keeping it grounded as your business changes

A grounded chatbot is only as current as the material it was built from. If your prices, policies or product range change and the source documents are not updated, the assistant will keep confidently repeating the old version, because as far as its retrieval step is concerned nothing has changed. This is a maintenance question as much as a build question: whoever keeps your chatbot's source content current is doing the work that actually keeps its answers correct, and it is worth being clear before launch about whose job that is.

In practice that means treating the assistant's source documents the same way you would treat the website itself — something that gets updated the same week a price or policy changes, not something revisited only when a customer points out it gave a wrong answer. A chatbot that quietly goes stale is a worse experience than no chatbot at all, because a visitor has no way to tell a confidently wrong answer from a correct one until it costs them something.

When not to buy one

A chatbot is not the right first move for every business. If your site gets very few visitors, the questions people ask are already answered clearly on the page, or you do not have enough of your own documented content to ground it in, a chatbot has little to retrieve from and will either answer too generally or decline too often to feel useful. In that situation the better spend is usually on the content itself, or on an honest audit of where automation would actually help before committing to a build.

• Ask it something your content does not cover — it should say so, not guess
• Ask it to connect you to a person — it should hand over cleanly
• Check whether answers cite the page or document they came from
• Review the gap report after launch, not just the conversations that went well
• If your own content is thin, fix that first — a chatbot cannot ground itself in pages that do not exist

A chatbot that can be wrong and knows it should say so is more useful, not less, than one that always sounds sure of itself.