Skip to main content
The fastest way to give your agent broad coverage of what your business does is to let it read your website.

​
Add a crawl

1

Open Knowledge

Sidebar → Knowledge → New source → Website crawl.
2

Paste the URL

Top-level domain works best: https://acme.com. You can also start from a sub-section (/docs) to scope the crawl.
3

Pick an agent

Choose which AI agent should have access. You can reassign later.
4

Start the crawl

Click Create. Keloa:
  1. Discovers the sitemap (or crawls outward from the URL).
  2. Fetches each page.
  3. Extracts clean text (no menus, no footers).
  4. Chunks it and indexes it.
Small sites finish in a minute; large ones take a few minutes to an hour.

​
What gets crawled

  • Pages on the same domain.
  • Content accessible to anonymous visitors (logged-in areas are not fetched).
  • HTML — not PDFs linked from pages. Upload those separately as file uploads.

​
What gets ignored

  • Navigation, footers, cookie banners.
  • JavaScript-rendered content that’s not prerendered.
  • Pages blocked by robots.txt or noindex.
  • Off-domain links.

​
Page limits

If your site has more pages than your cap, the crawl stops at the cap — we prioritise top-level pages first.
Starter is single-URL only. Full-site crawl (sitemap discovery + multi-page traversal) unlocks on Growth+. On Starter, paste the exact page you want indexed; the crawler fetches that one page and stops.

​
Watching progress

On the Knowledge list, the source status cycles queued → syncing → synced. Click the row to see individual pages, their status, and word count. Click View pages to inspect what was extracted.

​
Keeping content fresh

A crawl is a snapshot at that moment. Two ways to keep it current:
  • Recrawl manually — open the source → Recrawl. Re-fetches and re-indexes.
  • Schedule auto-recrawl — pick the strongest cadence your plan allows on each source.
The “manual syncs” budget is shared across every source and every kind of sync (recrawl + Shopify resync) per calendar month. When a page is removed from your site, it stays in the index until the next recrawl.

​
Tips

  • Start with your top-level domain. Scope down only if the crawl pulls in irrelevant content (blog posts, legal boilerplate).
  • After the crawl, chat with the agent (Test) and look for bad answers. Those point at missing or stale pages — patch with a Q&A pair rather than rewriting the site.
  • If a specific page shouldn’t be indexed, add it to your robots.txt.

​
Troubleshooting