Why Smarter Chunking Matters More Than Bigger LLMs

Introduction

In specialized fields, the goal is a smarter system, not just a bigger one. Scaling large language models gets a lot of attention, but real gains often come from fixing the backend, and in a Retrieval-Augmented Generation pipeline, that backend is largely chunking.

At Hoomanely, we kept running into irrelevant responses caused by poor chunking. Rather than reach for a bigger LLM, we fixed our chunking strategy. Switching to a dynamic approach preserved context and improved accuracy without raising costs. This post covers why smarter chunking beats scaling up, how we implemented it, and what changed as a result.

Why traditional chunking falls short in RAG

RAG pipelines are the backbone of chatbots working from a large knowledge base, in our case 2,796 veterinary PDFs. The process retrieves document chunks, feeds them to an LLM, and generates a response. Our initial setup had a few problems.

Noisy, inflexible chunking: we used fixed 512-token blocks with no overlap, which kept messy elements like image placeholders and headers in place, diluting content and losing context.

Embedding and indexing limits: our Neural Sparse Encoder with HNSW indexing was fast but struggled with the deep semantics in veterinary text, which produced noisy, inaccurate responses.

Mediocre NDCG@10 scores in our benchmarks: without proper context, the LLM was essentially guessing at connections. Scaling to a bigger LLM might have papered over these flaws, but it ignores the real cause: poor data preparation. As the dataset grows, inefficient chunking also raises storage costs and slows queries. We needed smarter chunking, not a bigger model.

A revamped RAG pipeline with dynamic chunking

We rebuilt our rechunking and reindexing strategy around document-processing best practices, using GCP Document AI for clean parsing, NV-Embed-v2 for embeddings, and improved indexing. The key change was moving from static to dynamic chunking, which adapts to content semantics rather than cutting at a fixed length.

The process now runs through pre-ingest parsing, structure extraction, dynamic chunking, filtering, and embedding and indexing. It scales well for our large veterinary corpus, and it's built around adaptability: dynamic chunking finds natural breaks instead of ignoring content boundaries the way static chunks do. It's also cost-effective and better suited to technical content, cutting hallucinations by giving the LLM cleaner, more contextual chunks.

Step-by-step breakdown

Pre-ingest stage, cleaning the source: we started with raw PDFs and used GCP Document AI to parse them, extracting structured JSON with paragraphs, headings, tables, forms, and figures, far better than simply flattening the text. Key steps included pulling fields like page number, paragraph text, layout confidence, table cells, and bounding boxes; cleaning by joining wrapped lines, normalizing Unicode (turning "μg" into "ug"), and removing headers and footers while keeping casing and units like mg/kg intact; and tagging quality, flagging anything with OCR confidence below 0.6. This processed our 1.44 million pages for around $2,200, covered by credits, and gave us clean input free of noise like base64 placeholders.

Structure extraction, building meaningful units: from the JSON, we pulled narratives (merging paragraphs under headings for flow, like a full disease-treatment section), tables (converted to Markdown or JSON, preserving dosage tables as structured data), forms (flattened into key-value pairs, like "Drug: Amoxicillin, Dosage: 10mg/kg"), and captions (attached to nearby text for context). That way, chunks come out as logical units instead of arbitrary slices.

Dynamic chunking, the core of the strategy: instead of fixed 512-token chunks with 10% overlap, we built a dynamic approach using semantic embeddings and adaptive boundaries, aiming for chunks that respect content semantics while staying within size limits for efficient retrieval. We defined window profiles like "512-cap" with a minimum of 380, a target of 480, and a max of 512 tokens, giving flexibility without wild variance. For unit preparation, we split paragraphs into sentences with SpaCy or blingfire, generated sentence embeddings with bge-small-en (fast, and it outperforms older models like all-MiniLM), and computed adjacent similarity for each sentence i as the cosine similarity between the mean embedding of the two sentences before it and the mean embedding of the two or three after it, which detects topic drift over short spans.

For candidate identification, we only evaluate cut points within the min-max range, delay cuts and boost overlap to 18-20% (up to 80 tokens) when a chunk falls below the minimum, and make atomic exceptions for tables or forms that start before the minimum, cutting early and rebalancing by borrowing up to 80 tokens from the prior chunk. Scoring favors topic drift, adds a structure boost for headings and paragraph ends, adds a cue boost for words like "However" or "Conclusion," and applies a length penalty favoring chunks near the 480-token target.

Local rebalancing uses a rolling three-chunk window to keep lengths within range, maximize total scores while limiting shifts to 80 tokens per boundary and 120 total, commit the oldest chunk, and slide forward, triggering a rebalance for short chunks by either borrowing from the previous one or increasing overlap. Overlap policy defaults to 10-15% (up to 80 tokens), drops to 5-10% at strong boundaries, rises to 15-20% at weak ones, and stays at 0% for tables and forms (replicating headers if a table gets split). Atomic rules keep tables intact, splitting rows only if they're oversized and always replicating headers, keep form overlap minimal at 5% or less, and attach captions to their figures. Every chunk also carries metadata: document ID, token length, whether it's a table, its boundary score, and whether it was rebalanced.

The boundary scoring formula for each candidate looks like this:

score = 0.6 * (1 - cosine_sim) + structure_boost + cue_boost - 0.2 * ((len_c - TARGET) / TARGET)**2

We treat a cosine below 0.78 as a strong shift and a drop greater than 0.15 as a sharp one, picking the highest-scoring candidate and falling back to paragraph ends if nothing qualifies. This keeps chunks semantically coherent, so a veterinary procedure doesn't get split awkwardly mid-explanation, while adaptive overlap prevents context loss between chunks. It costs more compute upfront than static chunking, mainly from the embeddings, but the retrieval quality more than makes up for it.

Pre-index filtering drops boilerplate, short text under 180 characters, poor OCR under 0.55 confidence, and empty tables, while promoting headings and numeric tables.

Embedding and indexing runs on NV-Embed-v2 (4096 dimensions, cosine similarity), which tops the MTEB benchmarks, is built on a Mistral-7B base, and is self-hostable. Indexing uses HNSW with M=32, ef_construction between 200 and 400, and ef_search at 128, with metadata including OCR confidence and section path.

Results

Switching to this pipeline, and dynamic chunking specifically, changed our RAG performance noticeably.

Cleaner chunks cut down on hallucinations, and dynamic boundaries improved recall for veterinary queries, for example retrieving full dosage tables intact instead of split across chunks. The approach scales to our 1.44 million pages, and the rebalancing step keeps results consistent.

Key takeaways

Smarter chunking isn't a minor tweak, it's a real force multiplier for RAG systems. Going dynamic showed us that fixing data prep beats scaling the LLM on its own. For engineers working on something similar: start with semantic embeddings for boundaries and adaptive overlap, it pays off in accuracy. For veterinary AI specifically, this is what makes answers reliable enough for real-world use. Before reaching for a bigger model, it's worth asking whether your chunking is actually good enough yet. This strategy came out of a lot of hands-on iteration, and it's worth trying against your own corpus.