Chunking
The process of splitting long documents into smaller passages before embedding them, so each piece can be retrieved and fitted into a model's context on its own.
Definition
Embedding models have input limits and, more importantly, a single vector cannot represent a 60-page document well. Chunking divides each document into passages that are small enough to embed meaningfully and to fit several of them into an LLM's context window at once. Each chunk is stored with a reference to its source and usually some metadata.
Strategies differ in how they find boundaries. Fixed-size chunking cuts every N tokens. Recursive splitting tries paragraphs first, then sentences, to keep natural units. Structural chunking follows headings, tables and list items in the document's own layout. Semantic chunking uses embeddings to detect topic shifts. Most add overlap between neighbours and attach the heading path to each chunk so it makes sense out of context.
Example: a 90-page employee handbook is split by section heading, with long sections further divided at paragraph breaks. Each chunk is stored as "Handbook › Leave › Parental leave › paragraph 3". A question about parental leave retrieves exactly those paragraphs, with their heading trail, rather than a random slice.
Why it matters for your business
- It is the most common cause of poor RAG answers: the fact was indexed, but cut in half or buried in noise.
- Chunk size drives index size, retrieval precision and token cost per query at the same time.
- Tables, PDFs and scanned documents need their own handling; generic splitters mangle them.
- Changing the strategy means re-indexing, so it should be decided with evals early.
How Teksmobile uses it
Document analysis and chunking strategy are the first technical steps in our document Q&A and enterprise RAG projects. We test candidate strategies against a retrieval eval set as part of our embeddings strategy service before the full corpus is indexed.
Related terms
Part of the Teksmobile AI Glossary. For longer reads, see our guides.
Frequently asked questions
What chunk size should we use?
There is no universal answer. Many systems land between a few hundred and a thousand tokens, but the right size depends on how your documents are written and what questions people ask. Test two or three sizes with retrieval evals.
What is chunk overlap?
Repeating the last few sentences of one chunk at the start of the next, so a fact that straddles a boundary is still retrievable in full. A modest overlap is common; a large one wastes index space.
Should chunks include metadata?
Yes. Storing the document title, section heading, date and permissions with each chunk improves both filtering and the context the LLM receives.