Website

Overview

The Website Knowledge Crawler crawls your public website or documentation portal, discovering linked pages, extracting body text, and refreshing embeddings whenever your site updates.

Crawler Capabilities & Sitemap Support

• Sitemap Discovery: Provide your sitemap.xml URL (e.g., https://mycompany.com/sitemap.xml) for instant batch discovery of all public pages.

• Deep Web Crawling: If no sitemap exists, provide a root URL and set a maximum crawl depth (e.g., Depth 3).

• Robots.txt Adherence: Respects robots.txt rules for the AsiystBot user agent.

Configuring Crawl Scope & Filters

Control which URLs are ingested using include and exclude glob patterns:

{
  "seedUrl": "https://docs.mycompany.com",
  "crawlDepth": 3,
  "includePatterns": [
    "/docs/**",
    "/api/**",
    "/guides/**"
  ],
  "excludePatterns": [
    "/changelog/archive/**",
    "/drafts/**",
    "*.pdf"
  ]
}

Handling JavaScript & Single Page Apps (SPAs)

For web apps built with Next.js, React, Vue, or Angular where content is rendered on the client:

• Enable Headless Rendering (Playwright) in crawler settings.

• The crawler boots a headless browser, waits for network idle (networkidle2), and captures the rendered DOM content.

Automated Re-Crawl Schedules

Keep your knowledge base synchronized with your publishing calendar:

• Daily: Best for fast-changing documentation and product blogs.

• Weekly (Default): Ideal for marketing websites and stable help centers.

• On-Demand Webhook: Trigger an instant re-crawl via your CMS publishing hook (e.g. Contentful, Sanity, WordPress).