Most RAG (Retrieval-Augmented Generation) implementations I've seen treat retrieval as an afterthought. They use a basic vector store, chunk documents arbitrarily, retrieve the top-5 results, and call it a day. This works fine for demos. In production, with real users asking ambiguous questions against a messy enterprise knowledge base, it falls apart within the first week.

I spent three months obsessing over retrieval while building Maha Maya. Here's what I learned about building retrieval that is fast enough to feel native and accurate enough to trust.

Why Standard Chunking Fails

The default advice is to chunk documents into fixed-size windows (512 or 1024 tokens) with some overlap. This is wrong for enterprise knowledge bases for a simple reason: business documents don't respect arbitrary token boundaries. A 512-token chunk of a legal contract might start mid-clause and end mid-sentence, giving the LLM a decontextualized fragment that's worse than useless.

Maha Maya uses semantic chunking instead. I compute sentence embeddings and use a cosine similarity threshold to detect topic boundaries:

def semantic_chunk(text: str, threshold: float = 0.75) -> list[str]:
    sentences = sent_tokenize(text)
    embeddings = embed_batch(sentences)
    
    chunks, current_chunk = [], [sentences[0]]
    
    for i in range(1, len(sentences)):
        similarity = cosine_similarity(
            embeddings[i-1].reshape(1,-1),
            embeddings[i].reshape(1,-1)
        )[0][0]
        
        if similarity < threshold:  # Topic boundary detected
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])
    
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    
    return chunks

HNSW Indexing for Sub-100ms Retrieval

Standard flat vector search (brute-force cosine similarity over all vectors) is O(n). At 10 million document chunks, this takes seconds. Maha Maya uses HNSW (Hierarchical Navigable Small World) indexing, which gives approximate nearest-neighbor search in O(log n) time:

import hnswlib

# Build index
index = hnswlib.Index(space='cosine', dim=768)
index.init_index(max_elements=10_000_000, ef_construction=200, M=16)
index.add_items(embeddings, ids=list(range(len(embeddings))))
index.set_ef(50)  # Query-time accuracy/speed tradeoff

# Query
def retrieve(query: str, top_k: int = 6) -> list[str]:
    q_embedding = embed(query)
    labels, distances = index.knn_query(q_embedding, k=top_k)
    return [document_store[label] for label in labels[0]]

With this setup, retrieval across 5 million chunks takes approximately 8ms on a single CPU core. Combined with Groq's inference speed (Llama 3.3 at ~750 tokens/second), the full query-to-response pipeline completes in under 100ms for most requests.

Supporting Marathi and Devanagari Scripts

Maha Maya was built specifically to handle multilingual enterprise knowledge bases, including documents in Marathi and Hindi (Devanagari script). Standard sentence-transformers struggle with Indic scripts because they're trained predominantly on English text. I fine-tuned a multilingual-E5 base model on a parallel corpus of Marathi-English business documents to ensure cross-lingual retrieval works correctly:

# Retrieve across languages — query in English, retrieve Marathi docs
results = retrieve("quarterly revenue targets for Maharashtra region")

# Returns relevant chunks even if they're written in Marathi:
# "महाराष्ट्र प्रदेशासाठी तिमाही महसूल लक्ष्य..."
# Similarity score: 0.89

Retrieval Evaluation: The Metrics That Matter

You cannot improve what you don't measure. I evaluate Maha Maya's retrieval quality using three metrics logged for every query in production:

  • Context Precision: What fraction of retrieved chunks are actually relevant to the query?
  • Context Recall: Did we retrieve all the chunks necessary to answer the question?
  • Answer Faithfulness: Is the LLM's final answer grounded in the retrieved context?
# RAGAS-style evaluation pipeline
def evaluate_retrieval(query, retrieved_chunks, ground_truth, llm_answer):
    precision = sum(is_relevant(c, ground_truth) for c in retrieved_chunks) / len(retrieved_chunks)
    recall    = sum(is_relevant(c, ground_truth) for c in retrieved_chunks) / len(ground_truth)
    faithful  = faithfulness_scorer.score(query, retrieved_chunks, llm_answer)
    
    return {"precision": precision, "recall": recall, "faithfulness": faithful}

The One Insight That Changed Everything

After all this engineering, the single biggest improvement to retrieval quality came from something deceptively simple: storing document metadata alongside embeddings and using it to pre-filter the search space. When a user asks about Q4 2025 sales data, filtering the vector index to only chunks from Q4 2025 documents before running HNSW search reduces the search space by 75% and eliminates an entire class of plausible-but-wrong retrievals.

Fast retrieval is a solved engineering problem once you pick the right index. Accurate retrieval requires understanding the structure of your documents and building retrieval logic that respects that structure. That's the harder, more interesting problem.