Knowledge Base
Knowledge Overview
Deep dive into the Asiyst retrieval-augmented generation (RAG) knowledge engine, vector indexing, and grounding architectures.
Overview
The Asiyst Knowledge Engine is an enterprise Retrieval-Augmented Generation (RAG) platform that grounds AI assistant responses in your actual company documentation, support articles, product catalogs, and GitHub repositories. By coupling dense vector semantic search with sparse BM25 keyword matching, Asiyst ensures assistants deliver strictly accurate, verifiable answers with clickable source citations.
The RAG (Retrieval-Augmented Generation) Architecture
┌─────────────────────────────────────────────────────────────┐
│ Content Ingestion │
│ Websites │ PDF / Markdown │ GitHub │ Custom URLs │
└──────────────────────────────┬──────────────────────────────┘
│ (Parsing & Semantic Chunking)
▼
┌─────────────────────────────────────────────────────────────┐
│ Vector Indexing Store │
│ Dense Embeddings (1536-dim) + Sparse BM25 Keyword Index │
└──────────────────────────────┬──────────────────────────────┘
│ (Hybrid Cosine + Keyword Query)
▼
┌─────────────────────────────────────────────────────────────┐
│ Context Injection & LLM │
│ Top-K Grounded Chunks + System Instructions ──▶ LLM │
└─────────────────────────────────────────────────────────────┘
- Ingestion & Parsing: Raw files and web pages are parsed, stripped of boilerplate HTML navigation, and decomposed into semantically coherent chunks (typically 500 tokens with 10% overlap).
- Dense Vectorization: Chunks are processed through embedding models to generate high-dimensional vectors representing semantic meaning.
- Hybrid Search Retrieval: When a visitor asks a question, Asiyst searches the vector index for conceptual similarity and the keyword index for exact acronyms or model numbers.
- Context Injection: The top 3–5 most relevant passages are placed inside the system prompt as immutable grounding evidence.
- Grounded Generation: The LLM synthesizes a concise, natural response and appends interactive citation tags.
Dense Embeddings & Vector Storage
Asiyst stores vector representations in high-performance vector databases optimized for sub-15ms nearest-neighbor search (HNSW index). Vectors are isolated per project, ensuring zero cross-tenant contamination.
Multi-Source Aggregation
You can connect diverse source types into a unified knowledge collection:
- Public Websites & Sitemaps: Automated crawling of your live marketing site, help center, or API docs.
- Documents & Manuals: PDF whitepapers, Markdown guides, plain text FAQ dumps, and DOCX files.
- GitHub Repositories: Source code, README files, doc directories, and release notes synchronized on commit.
- Live URL Lists: Discrete public or authenticated web endpoints.
Citation & Verifiable Truth
Every factual statement generated by the assistant is linked to an underlying text excerpt. Visitors can hover over or tap citation badges to view the exact excerpt and verify the information for themselves, fostering trust.
Lifecycle of a Knowledge Document
[Upload / Crawl] ──▶ [Parsing] ──▶ [Chunking] ──▶ [Embedding] ──▶ [Indexed / Ready]
│
[Re-sync on update]