Indexing

Overview

Indexing is the process of converting raw text chunks into dense vector representations and organizing them into an approximate nearest neighbor (ANN) index for real-time querying.

Vector Indexing Mechanics

• Embedding Model: Text chunks are embedded into 1536-dimensional vectors using state-of-the-art embedding models.

• Index Type: Hierarchical Navigable Small World (HNSW) graph index.

• Distance Metric: Cosine similarity ($1.0$ = identical semantic meaning, $0.0$ = completely unrelated).

Hybrid Search: Sparse BM25 + Dense Vectors

Pure vector search can occasionally miss exact keyword identifiers (e.g., error codes like ERR_402_AUTH or specific product SKUs). Asiyst employs Reciprocal Rank Fusion (RRF) to combine:

1. Dense Vector Search: Captures natural language synonyms and concepts.

1. Sparse BM25 Search: Matches exact keywords, codes, and acronyms.

$$\text{RRF Score} = \sum_{m \in \{\text{Dense}, \text{Sparse}\}} \frac{1}{k + \text{Rank}_m(d)}$$

Index Rebuilding & Pruning

When sources are deleted or modified, old vectors are pruned immediately. Rebuilding an entire index of 10,000 pages takes under 45 seconds.