Adding a Website Source
Click Add Source → Website
Enter the root URL of the website you want to scrape (e.g.
https://docs.example.com).Discover pages
FIFE.BOT crawls the sitemap and lists all discovered pages. You’ll see:
- Page URL
- Page title (if available)
- Estimated content size
Adding More Pages Later
You can add more pages to an existing website source at any time. Open the source, click Add pages, and select additional URLs from the sitemap.Auto-Reindex
Website sources are automatically re-scraped on a schedule to keep your knowledge base up to date. The scrape worker runs every 30 minutes and processes sources based on their configured reindex interval.How many knowledge sources you can attach per chatbot depends on your plan (see Billing & Plans). Website scraping is one type of source; scheduled re-index keeps pages up to date.
Routing Instructions
You can add routing instructions to a website source to give the AI extra context. For example:“This source contains our product documentation. When referencing it, always include the relevant product version number.”
Processing Status
| Status | Meaning |
|---|---|
| Ready | Page is indexed and searchable |
| Processing | Page is being scraped and embedded |
| Error | Scraping failed — hover to see the error message, click to retry |
Troubleshooting
| Issue | Solution |
|---|---|
| No pages discovered | Check if the site has a sitemap.xml. Some SPAs don’t expose one. |
| Page scrape failed | The page might be behind authentication, have anti-bot protection, or return errors. |
| Content seems outdated | Check the last reindex date. You can manually trigger a resync. |
| Too many pages | Select only the most relevant pages. You don’t need to scrape your entire site. |