> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fife.bot/llms.txt
> Use this file to discover all available pages before exploring further.

# Websites

> Scrape and index web pages to feed your chatbot with up-to-date website content.

Website scraping lets you turn any website into chatbot knowledge. FIFE.BOT uses [Firecrawl](https://firecrawl.dev) to crawl pages, extract clean text, and index it for retrieval.

## Adding a Website Source

<Steps>
  <Step title="Open Knowledge Base tab">
    Go to your chatbot → **Knowledge Base** tab.
  </Step>

  <Step title="Click Add Source → Website">
    Enter the root URL of the website you want to scrape (e.g. `https://docs.example.com`).
  </Step>

  <Step title="Discover pages">
    FIFE.BOT crawls the sitemap and lists all discovered pages. You'll see:

    * Page URL
    * Page title (if available)
    * Estimated content size
  </Step>

  <Step title="Select pages">
    Check the pages you want to include. You can select all or pick individual pages.
  </Step>

  <Step title="Scrape">
    Click **Scrape selected**. Each page is:

    1. Fetched via Firecrawl
    2. Cleaned (HTML → text)
    3. Split into chunks
    4. Embedded as vectors
    5. Indexed for search
  </Step>
</Steps>

## Adding More Pages Later

You can add more pages to an existing website source at any time. Open the source, click **Add pages**, and select additional URLs from the sitemap.

## Auto-Reindex

Website sources are **automatically re-scraped on a schedule** to keep your knowledge base up to date. The scrape worker runs every 30 minutes and processes sources based on their configured reindex interval.

<Info>
  How many **knowledge sources** you can attach per chatbot depends on your plan (see [Billing & Plans](/a-14)). Website scraping is one type of source; scheduled re-index keeps pages up to date.
</Info>

## Routing Instructions

You can add routing instructions to a website source to give the AI extra context. For example:

> "This source contains our product documentation. When referencing it, always include the relevant product version number."

## Processing Status

| Status     | Meaning                                                          |
| ---------- | ---------------------------------------------------------------- |
| Ready      | Page is indexed and searchable                                   |
| Processing | Page is being scraped and embedded                               |
| Error      | Scraping failed — hover to see the error message, click to retry |

## Troubleshooting

| Issue                  | Solution                                                                             |
| ---------------------- | ------------------------------------------------------------------------------------ |
| No pages discovered    | Check if the site has a sitemap.xml. Some SPAs don't expose one.                     |
| Page scrape failed     | The page might be behind authentication, have anti-bot protection, or return errors. |
| Content seems outdated | Check the last reindex date. You can manually trigger a resync.                      |
| Too many pages         | Select only the most relevant pages. You don't need to scrape your entire site.      |