Key takeaway: When retrieval returns irrelevant passages, the usual cause is a chunking strategy that severed the context the passage needed to be meaningful.
The Default That Breaks Everything
Nearly every RAG tutorial starts with fixed-size chunking: split at 512 tokens, overlap by 50, embed each piece. It is simple, fast and wrong for most real documents.
Fixed-size splitting cuts through the middle of tables, separates a heading from the section it introduces, and orphans a pronoun from its antecedent. A chunk reading “This limit applies to all requests in the tier” is useless — the retriever cannot match it and the model cannot use it, because “this limit” and “the tier” were defined three hundred tokens earlier.
Structure Beats Size
The better mental model is that a chunk should be a self-contained answer to a plausible question. That means splitting along semantic boundaries the document already provides.
| Strategy | Retrieval quality | Best suited to |
|---|---|---|
| Fixed 512 tokens | Poor | Uniform prose, no structure |
| Split on headings | Good | Documentation, wikis, manuals |
| Split on headings + prepend breadcrumb | Very good | Technical docs, policies |
| One row per chunk with header context | Very good | Tables, reference data |
| Whole document | Poor | Anything over ~1,500 tokens |
The single highest-return change in most systems is prepending the document title and heading path to every chunk before embedding. A chunk that begins “Billing > Rate Limits > Enterprise Tier” carries the context needed for both retrieval matching and generation, at a cost of perhaps twenty tokens.
Handling the Awkward Cases
Tables break under prose chunking. Split them row-wise and repeat the header row in each chunk, or serialise each row as a sentence. A row that reads “Enterprise tier: 10,000 requests per minute, $2,000 per month” retrieves well; a naked cell containing 10,000 does not.
Long sections that exceed your target still need splitting. Split at paragraph boundaries rather than token counts, and carry the heading breadcrumb into every piece.
Code blocks should stay intact even when oversized. A half function is worse than no function, and the extra tokens cost less than a wrong answer.
Verifying It Improved
Chunking changes are easy to make and easy to fool yourself about. Build a fixed evaluation set of thirty to fifty real questions with known correct source passages, then measure recall at k before and after every change. Without that harness you are guessing, and chunking is exactly the area where intuition misleads.
Watch for the specific failure of retrieving the right document but the wrong section. That symptom points squarely at chunk boundaries rather than at the embedding model everyone reaches to replace first.
The Bottom Line
Before swapping embedding models or migrating vector databases, print twenty of your actual chunks and read them as a stranger would. If they do not make sense in isolation, no retrieval system can rescue them. Chunking is preprocessing, it is unglamorous, and it determines the ceiling on everything downstream.
