To train an AI chatbot on your website, you do not actually train anything. The model never changes. What happens is closer to filing than teaching: your pages get fetched, cleaned up, split into chunks, and stored in an index. When a visitor asks something, the bot retrieves the most relevant chunks and answers from them, quoting your content instead of improvising about it. The mechanism is called retrieval-augmented generation, and we wrote a plain-English explainer on RAG if you want the theory. This post is the practice: getting your website into that index without also getting the junk.
The distinction matters because it changes what training costs and what it fixes. There is no model run to wait for. Updating the bot means updating the index, which takes minutes. It also means answer quality is decided almost entirely by what goes into the index. Which is exactly the part most tools rush past.
One more expectation to set: the bot does not learn from its own conversations unless you tell it to. Chats do not silently become knowledge, and that is a feature; you do not want one customer's confusion becoming another customer's answer. Corrections happen deliberately, by fixing the source content or adding a curated answer, and the better tools turn that into a short loop from a flagged conversation to a fix.
What blind indexing gets wrong
The standard flow is one field and one button. Paste your URL, click train, watch a progress bar, receive a cheerful "your chatbot is ready" screen. It feels like magic, mostly because you have no idea what just happened.
Here is what tends to be in that index. Your navigation menu, repeated on every page crawled. Your footer, likewise. The cookie banner, two hundred times. Duplicate pages that differ only by a tracking parameter or a trailing slash. A blog post from 2021 quoting prices you no longer charge. And your marketing copy, which the bot will now recite to customers as if it were policy. A landing page says something breezy like "easy returns, no questions asked"; the actual policy says 14 days, unworn, original packaging. Guess which one the bot quotes when both sit in the index with equal standing.
Duplicates deserve a special mention because the damage is quiet. Retrieval pulls a handful of the closest chunks for each question, maybe five. If your shipping page got indexed three times under three URLs, those five slots fill up with three copies of the same paragraph, and the answer loses the context it would have had from the returns page or the FAQ. The bot gets more confident and less informed at the same time.
The deeper problem is visibility. Ask what got indexed and you get a page count. Ask which pages, and which parts of those pages, and you get a shrug. You cannot fix a knowledge base you cannot see, and you will not find out what is in it until a customer does.
Scan before you index
The fix is boring and it works: separate the scan from the index.
First, crawl the site and extract content from every page found, but store nothing in the knowledge base yet. Chunk each page exactly the way it would be chunked for real, count the results, and present the whole thing as a review table: every URL, its exact chunk count, grouped by path so that 80 blog posts collapse into one row you can deselect together. Then you choose what goes in. Only after you confirm does anything get embedded and indexed.
This is how we built the training flow in Hey Support, and I would argue for the pattern even if you use a different tool. Two minutes of review beats a month of wondering why the bot keeps citing your careers page.
Exact counts matter more than they look. Some tools estimate scope in bytes or raw page tallies, which mostly measures HTML weight rather than usable text; a template-heavy page can be enormous on the wire and nearly empty of content. A chunk count produced by the same code that will do the real indexing is a precise preview: what the review table shows is what lands in the index, number for number. It also keeps plan limits honest, because you can see what you are spending capacity on before you spend it.
Say the crawl finds 300 URLs. In review, maybe 60 of them carry answers customers actually need: product pages, shipping and returns, docs, pricing, contact. The rest is tag archives, pagination, press mentions, and a decade of blog posts. Deselecting the 240 takes two minutes and improves every answer the bot ever gives, because retrieval now picks from a clean pool instead of fishing in the archive.
A rule of thumb for the review pass: if you would not paste the page into a live chat with a customer, do not index it. Careers, press, legal boilerplate, the founder interview from 2020. All fine pages. None of them are support knowledge.
Extraction details that decide answer quality
With the right pages selected, a handful of unglamorous details decide whether answers come out precise or mushy.
Strip the chrome before extracting text. Navigation, header, footer, sidebar. If they survive, every chunk on every page opens with the same menu links, retrieval starts matching on menu text, and answers arrive padded with "Home About Contact". This sounds too obvious to state. Test any bot that responds to a product question with a list of page names and you will find chrome in its index.
Keep the heading hierarchy. A paragraph about customs fees means one thing under "Shipping, international orders" and something else under "Returns". Good extraction records each chunk's heading path, H1 then H2 then H3, and stores it with the text. The bot then answers with context attached instead of guessing which section a floating paragraph came from.
Chunk size is the least intuitive part, so here is the plain version. Chunks of a thousand-odd characters, with a little overlap between neighbors, tend to work well: large enough that a policy clause keeps its surrounding conditions, small enough that retrieval can be choosy about what it pulls. The overlap exists so a sentence that straddles a boundary appears whole in at least one chunk instead of arriving cut mid-thought in both.
Preserve links as text plus URL. Before stripping tags, convert every link into its visible text with the address alongside, so "see our size guide (yourstore.com/size-guide)" survives into the chunk. Now the bot can hand visitors real links. Pair that with a hard rule: only share URLs that exist in the indexed content, never compose one. Language models will happily produce plausible-looking URLs that lead nowhere; the mechanism behind that is covered in why AI chatbots make things up. A bot that links to a page that does not exist is worse than a bot that shares no link at all.
Keeping the knowledge fresh
An index is a snapshot. Your website is not.
Prices change, policies get revised, products get discontinued, and every change opens a gap between what your site says and what your bot says. Manual retraining closes that gap exactly as reliably as you remember to do it, which for most teams means never, or in a hurry after a customer got a stale answer.
Put it on a schedule instead. A refresh job re-crawls the indexed pages, hashes the extracted content of each one, and compares against the stored hash. Unchanged pages get skipped, with no re-embedding and no cost. Changed pages get re-chunked and re-indexed. Pages that fail to fetch keep their existing knowledge rather than being wiped by a timeout, a rule you only think to write after a bad crawl has erased good content once.
In Hey Support this is a setting rather than a chore: weekly refresh on Growth, daily on Scale, with hash-skipping doing the economics. A 300-page site where three pages changed costs three pages to refresh. Uploaded files, pasted text, and Q&A pairs sit outside the cycle, since nothing can re-crawl a PDF you exported in March; put a quarterly reminder on those. Whatever tool you use, the questions for the crawled part are the same. Does it refresh on a schedule? Does it skip unchanged content? Can a failed crawl destroy good knowledge?
When crawls fail (and what to do)
Crawls fail. Any tool claiming a 100 percent success rate is describing websites it has not met yet. The common failure modes are worth knowing because the fixes differ.
Bot protection is the big one. Cloudflare and similar services challenge unfamiliar crawlers, and sometimes the challenge page returns a normal 200 status, so a naive crawler happily indexes "checking your browser" as knowledge. A good tool detects the interstitial and reports it as a block instead of a success.
Rate limiting hits when a crawler moves too fast; the polite fix is slower crawling with retries, and the impatient fix is in the next paragraph. JavaScript-only pages are the sneaky one: if your content renders client-side, a plain fetch sees an empty shell. Then there is the ordinary debris of any site, 404s from stale internal links, non-HTML files, pages with no real text on them.
Whichever it is, the tool should tell you which page failed and why, per page, with a specific reason. "Failed to crawl site" as a blanket message is a support ticket generator.
Then route around it. If discovery is the problem, paste your sitemap URL and skip link-following entirely. If fetching is the problem, export the content and upload files, since PDFs and text documents index fine, or paste the text straight in. For the ten questions that matter most, add them as Q&A pairs, which need no crawling at all. The content is what matters; how it arrives is just transport.
How to prep your site before training
You can raise answer quality before touching any tool. An hour of prep pays off in every answer afterwards.
- Put real answers on crawlable URLs. A returns policy that lives in a PDF attached to order emails, or inside a popup modal, might as well be private. Give it a page.
- Keep one canonical FAQ page. Five scattered partial FAQs that mildly disagree will produce a bot that mildly disagrees with itself.
- Title pages like you mean it. "Shipping and returns" retrieves well. "Info" retrieves nothing.
- Retire or noindex what you would deselect anyway, like the 2019 pricing announcement and the discontinued product line.
- If you run aggressive bot protection, know that before you crawl, and keep the sitemap handy as a fallback.
- Write down the answer you keep sending by email. If it is not on the site, the bot cannot give it, and frankly neither can your search results.
Once the index is live, test for honesty before you test for coverage. Ask questions your content genuinely does not answer and watch what comes back; the right response is a plain "I don't know" with a route to a person, and anything more creative is a warning sign. Then ask questions the content does answer and check that the citations point where you expect. Twenty minutes of adversarial poking tells you more than any setup wizard will.
The pattern behind all of this is the same. The bot is only as honest as its inputs. Retrieval does not create good content; it can only find what you gave it and quote it accurately. A chatbot with clean, current knowledge is downstream of a team that looked at what went in.
If you want to watch the scan-and-review step run on your own site, signing up is free: 50 conversations a month, and the review table works the same on every plan. The full setup walkthrough lives in the docs.
