🧠 AI with Python – ✂️ RAG Systems: Document Chunking Strategies 🐍

A document may contain the answer while retrieval still misses the evidence. One reason is how the document was split before embedding.

Chunking divides documents into units that can be embedded and retrieved. A useful chunk contains enough context to make sense while remaining focused and within the model's input limit.

✂️ Three Strategies to Compare

This lab uses fictional Markdown policies and compares:

The character counts make the example easy to inspect. They are demonstration settings, not recommended production defaults.

📚 Preserve the Source While Splitting

A basic window advances by size - overlap:

for start in range(0, len(text), size - overlap):
    end = min(start + size, len(text))
    chunk_text = text[start:end]
    if end == len(text):
        break

The notebook validates 0 <= overlap < size and attaches a document ID, title, section, version hash, and character offsets to each chunk. An assertion checks that every chunk still matches its original source slice.

The section splitter recognises Markdown level-two headings. Oversized sections fall back to bounded windows, so even this strategy can split a sentence. Tables, PDFs, and complex layouts need appropriate parsing rather than assuming this small splitter handles every document.

🔍 Embed and Retrieve Consistently

Use the same model for documents and questions:

vectors = model.encode(
    [chunk["text"] for chunk in chunks],
    normalize_embeddings=True,
    convert_to_numpy=True,
)
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(np.asarray(vectors, dtype="float32"))

For normalised vectors, inner-product search ranks by cosine similarity. FAISS metric documentation

Check token lengths before embedding. Characters and tokens are different units, and text beyond an embedding model's sequence limit may be truncated. The notebook checks each chunk against the loaded model's limit. Sentence Transformers input limits

✅ Evaluate the Evidence

We use four questions with known evidence sentences. A hit requires a top-two result from the correct document containing the complete sentence.

In this example, overlap adds 600 indexed characters compared with the fixed windows. Section chunks omit the heading syntax from the embedded body while retaining section names as metadata.

This is a small, strict diagnostic. Different strategies return different amounts of text, and exact sentence containment does not capture every form of relevant evidence. The results do not establish a universal winner.

📌 Key Takeaways

• Chunk boundaries affect the evidence available to retrieval.
• Overlap repeats context and increases indexed text.
• Structure-aware splitting still needs a size limit and appropriate parsing.
• Preserve source metadata and check model token limits.
• Tune on representative questions, inspecting both retrieval misses and unnecessary context.