Knowledge

Website

Scan a whole site, paste a sitemap or add a single page, then choose exactly which pages your agent learns from

Most agents get most of their knowledge from a website. Open Knowledge, select Add knowledge, then Website.

The add website panel. A URL with a path is entered, so three scope options are offered: Entire site, This section only and Just this page, with a row of common paths to exclude and a Scan site button.
This section only appears because the URL has a path. The suggested exclude chips below are one tap each.

Choose how much to read

Enter your homepage

Pick Entire site to read every page we can find, from your sitemap or by following links. If the URL has a path, This section only reads only the pages under it, so yoursite.com/docs reads your docs and nothing else.

Skip what you do not want

Add exclude chips for sections like /blog or /legal. They apply to this scan only; to skip pages for good, add an exclude rule.

Scan, then confirm

We scan first and read pages while the tab is open. A small site is learned with nothing to confirm. A large site shows what we found before anything is added, so you choose what to keep.

Paste the sitemap address

Enter the URL of your sitemap.xml. We read the list of pages from it instead of following links.

Review and confirm

From here it works like a whole-site scan: a small site is learned straight away, a large site asks first.

Enter the page address

Pick Just this page.

It is learned straight away

There is no scan step. Use it for one new article or a page a scan missed.

Scanning is free and never counts against your plan. Only the pages you keep do.

Large sites ask first

If your site is bigger than one scan reads, or would not fit your plan, we stop and show you:

  • The real number. "We found 1,181 pages", not a capped count.
  • What fits. We read a few pages to measure how much text this site puts on a page, then say roughly how many your plan holds.
  • A starting selection. Pages visitors ask about (/faqs, /about, /contact) come before the thousandth product listing.
  • Your call. Pages appear as a tree of folders; filter by path and tick or untick a folder at once.

Your website's pages

Each website has its own card under Sources on the Knowledge page. Choose Pages to see every page we know of, as a tree, with filters: All, Learned, Not added, Couldn't read and Excluded.

FilterWhat is in itWhat you can do
LearnedThe pages your agent uses, with how much text each holdsChoose Select, tick pages or a folder, then Remove. Open a page to see what your agent learned, Read it again, or edit its Suggested questions.
Not addedPages we found that your agent has not read, marked Suggested, Visitors asked about this or NewChoose Select, tick pages, then Add, or Don't suggest to move them out of the way. Add N suggested adds every suggested page at once.
Couldn't readPages the last sync could not reach, with the reason (shown only when there are any)Try all again, or remove a page that is gone from your site.
ExcludedPages you removed and pages that match an exclude ruleAdd and edit exclude rules. Include moves a page back to Not added.

Exclude rules

On the Excluded filter, choose Add rule for pages your agent should never use.

RuleExampleMatches
Starts with/blog/blog and every page under it, like /blog/any-post
ContainsinternalAny page with that text in its path
Exact page/old-pricingThat one page

As you type, the rule shows how many pages it matches. Rules apply to every website of the agent and to every future scan, suggestion and sync. If a new rule matches pages your agent already learned from, you are asked first; saving it removes them.

Next steps