Knowledge Base
Processing
Content sanitization, HTML boilerplate removal, Markdown normalization, and semantic chunking algorithms.
Overview
Before text is embedded, it undergoes automated cleaning and semantic chunking to ensure the AI model receives clean, relevant passages without irrelevant header/footer navigation junk.
Cleaning & Normalization Pipeline
- Boilerplate Stripping: Removes site navigation headers, footer links, cookie notices, and ads.
- Markdown Normalization: Standardizes headings, code formatting, and bulleted lists.
- Link Resolution: Converts relative links (
/docs/auth) into absolute URLs (https://asiyst.com/docs/auth).
Semantic Chunking Algorithms
- Paragraph Boundary Aware: Chunks never break mid-sentence or mid-code-block.
- Optimal Size: Default chunk size is 400–600 tokens with a 50-token overlapping window to preserve continuity between adjacent chunks.
Metadata Tag Enrichment
Each chunk is enriched with contextual metadata:
- Document title and breadcrumb path
- Section heading hierarchy (e.g.
# Installation > ## React > ### App Router)
- Last updated timestamp