Skip to main content

Train on Your Website

Use website training to let an AI chatbot agent learn from web pages. You can add one URL directly or use the crawler to discover pages from a website.

Add One URL Directly​

Open the agent's Training page and enter a website URL in the Enter URL address field.

Click the green + button to add the URL as a training source.

Website URL added as a training source

After the URL is added, click Start training to process it.

Use the URL row actions when you need to adjust preprocessing settings or remove the URL before training.

URL row with document training preprocessing options

Crawl a Website​

Use the crawler when you want Chatislav to find multiple pages from the same website.

Enter the base URL, then click the crawler icon next to the URL field.

Website URL field with crawler action

Configure the Crawler​

The Website Crawler modal controls how Chatislav discovers pages.

Website crawler settings modal

Crawler options include:

  • Use sitemap: use the website sitemap when available.
  • Sitemap: provide a sitemap URL if you want to use a specific sitemap.
  • Crawl: crawl linked pages from the starting URL.
  • URL filter: limit discovery to matching URLs.
  • Cookie: provide a cookie when the site requires one.
  • Max pages: limit how many pages the crawler should collect.

Click Start to run the crawler.

Select Found URLs​

After crawling, Chatislav shows the found URLs. Review the list and choose which pages to add.

Found URLs selection list after website crawl

The results screen includes:

  • Search for filtering discovered URLs.
  • Select / deselect all for bulk selection.
  • From and to fields for selecting a range.
  • Exclude files to avoid adding file URLs.
  • Per-URL toggles for individual selection.

Click Add after selecting the URLs you want to train on.

Start Training​

Selected URLs appear in the training source list. Confirm the data language, adjust preprocessing if needed, then click Start training.

After training starts, website URLs use the same document table statuses as uploaded files:

  • In progress while Chatislav is processing the page.
  • Trained when the page is ready for retrieval.
  • Failed if the page could not be processed.

Best Practices​

  • Start with important pages first, such as product, support, pricing, or FAQ pages.
  • Use the crawler when you need to add many pages from the same site.
  • Use URL filters to avoid unrelated sections.
  • Exclude files when you only want regular web pages.
  • Keep the page limit reasonable, then add more pages after checking the first results.