Our RAG Demo Worked. Production Didn't. So We Rebuilt Ingestion.

Thousands of files in. Millions of chunks out. The demo was the easy part.

A few months ago, a client came to us with a requirement that sounds simple in a kickoff meeting: build RAG over our knowledge corpus.

Then we opened the corpus.

Thousands of files. Long reports, dense tables, code, structured documents — all jumbled together. Millions of chunks once processed. This was not a "drop ten PDFs into a vector database" problem. It was an enterprise knowledge system, and it had to be trustworthy enough to put in front of users.

We built v1 the way most tutorials tell you to. It looked great in the demo.

Then real questions arrived.

The short version: At scale, RAG accuracy isn't something you bolt on at query time with a better prompt. It's built into how you ingest, structure, and index everything — before the first question ever gets asked. This is the story of the seven ingestion changes that took our client's system from demo-grade to trusted.

The demo lied to us (politely)

Version 1 followed the standard recipe:

1. Split every document into fixed-size chunks.
2. Embed the chunks.
3. Search, retrieve, generate.

For a demo, that recipe is perfect. The documents are few, the questions are friendly, and somebody on the team already knows which paragraph the answer lives in.

Production was less friendly:

• Answers cited chunks that started mid-sentence. The evidence was technically retrieved — it just began halfway through a thought, so neither the model nor the reader could fully trust it.
• A question about performance numbers returned a paragraph instead of the table that actually contained the answer.
• Tables came back flattened. Rows and columns dissolved into a stream of text. The numbers survived; their meaning didn't.
• Broad questions got random fragments. Ask "what's in this document?" and chunk-level retrieval has nothing at document scope to return — so it returns whatever fragment scores highest, which is close to a guess.

None of these were dramatic failures. No crashes, no empty results, no obvious nonsense. Each one was a small accuracy leak.

At this scale, small leaks add up. The client's feedback was blunt: they couldn't put the system in front of users if they couldn't trust it.

So we stopped tweaking prompts — and went back to ingestion.

The system we built instead

Ingestion (top) does the heavy lifting once. Retrieval (bottom) stays simple, parallel, and tenant-scoped. Open the interactive version →

1. We stopped treating every document the same

A report, a table, and a code file are not the same kind of object. Fixed-size chunking pretends they are — and destroys exactly the structure that makes each one intelligible.

Now we parse each file into its real structure — headings, sections, tables, code blocks — and chunk along those boundaries:

• Code fences are respected, so a # inside a code block is never mistaken for a heading.
• Sections stay whole wherever they reasonably can.
• Oversized sections split at word boundaries, so nothing is ever cut mid-sentence.

The chunk the model cites is now the chunk a human would have highlighted: complete, bounded, and recognisable.

2. We gave every chunk an address

A retrieved fragment floating free of its document is hard to verify. So every chunk now carries its full heading path —

Report › Performance › Benchmarks

— plus its section and parent indices within that path.

That one change pays off three times. Results know exactly where they came from, so citations point at a place a reader can open and check. The model gets the surrounding context ("this number lives under Benchmarks, not under Targets") for free. And when an answer is wrong, the team can debug it: they can see precisely which addressed chunk produced it, instead of hunting through a corpus of anonymous fragments.

3. Tables got their own treatment

Flattening a table into prose is a lossy compression that throws away the only thing a table is for: the alignment between rows and columns.

Now every table is its own chunk, stored with:

• the full Markdown preserved, so structure survives into the answer, and
• a clean CSV copy, so the data stays machine-readable and exact.

Ask for performance numbers and you get the table — columns lined up, values in their cells, readable by both the model and the person checking its work.

4. We made tables findable by meaning, not just cell values

Keeping tables intact solved readability. It didn't solve findability.

Real questions rarely use the words inside the cells. Nobody asks "show me the row where the third column reads 42 ms." They ask "how much faster did it get?" — and the words faster, before, and after may appear in no cell at all.

So an LLM reads each table at ingestion time and writes a short semantic description of what it actually compares — something like:

"Compares latency before and after optimisation across 5 metrics."

That description is indexed alongside the table. The question and the table finally meet in the same vocabulary: meaning. Semantic table search is the difference between a corpus that contains the answer and a system that can find it.

5. Every file got a summary vector

"What's in this document?" is a legitimate, common question — and pure chunk retrieval cannot answer it. Every chunk is a detail; the question is about the whole.

So each file also produces one summary vector at ingestion. Broad, document-level questions now retrieve a document-level answer, while specific questions still land on the exact section. Matching the scope of the retrieval unit to the scope of the question turned one of v1's most embarrassing failure modes into an ordinary query.

6. Prose and tables stopped competing unfairly

In a single shared index, prose and tables compete on prose's terms. A long, wordy paragraph accumulates similarity signals a compact table never will — so the paragraph that mentions performance beats the table that contains it.

We separated them:

• Prose and tables live in separate indexes (with summaries alongside prose retrieval).
• They are queried in parallel for every question.
• Results merge into one list ranked purely by relevance.

A strong table match beats a weak paragraph. A strong paragraph beats a weak table. The format that actually answers the question wins — which is the only ranking rule users ever cared about.

7. Every tenant got hard isolation

This is a multi-tenant system, and "we filter by tenant in the query" is a promise that one forgotten filter can break.

Instead, each tenant has its own partition in the vector store. Every write lands in one partition; every search reads from one partition. Isolation holds at the storage layer — not because application code remembered to enforce it, but because there is no shared space for data to leak into.

For a system holding client knowledge corpora, that distinction is the difference between an architecture you can defend and one you merely hope about.

Underneath: ingestion you don't have to babysit

The seven changes above get the attention, but none of them matter if ingestion itself is fragile. At millions of chunks, it also has to be boring:

• Background processing with live progress — large corpora ingest as tracked jobs, not as a request somebody waits on.
• Idempotent re-ingests — when a document is re-processed, its old vectors are deleted before new ones land. Re-running ingestion is always safe; the index never accumulates ghosts of previous versions.
• Batched embedding and upsert calls — work is grouped to stay within API rate limits, so a full-corpus ingest finishes instead of throttling halfway through.

Reliability here is unglamorous — and it's what makes the accuracy work deployable rather than theoretical.

v1 vs. v2, honestly

Failure in the field	Naive v1 (fixed-size)	Structure-aware v2
Citations start mid-sentence	Fixed windows cut anywhere	Chunks follow sections; splits at word boundaries
Code breaks parsing	# in code read as a heading	Code fences respected during parsing
Answers can't be located	Anonymous fragments	Full heading path + section/parent indices on every chunk
Tables return as flattened text	Tables split and dissolved into prose	One chunk per table, Markdown + CSV preserved
"How much faster?" finds nothing	Only cell values indexed	LLM-written semantic description indexed with the table
"What's in this document?" gets fragments	No document-scope unit exists	One summary vector per file
Paragraphs crowd out tables	One index, one unfair contest	Separate indexes, parallel query, merged by relevance
Tenant data could mix	Shared space, filter-dependent	Hard per-tenant partitions in the vector store
Re-ingesting corrupts the index	Duplicates and stale vectors pile up	Delete-before-write; idempotent by construction

What we'd check before trusting any RAG system

If you're evaluating your own pipeline — or a vendor's demo — these questions surface most of the leaks we hit:

1. Open a retrieved chunk at random. Does it start and end at a real boundary, or mid-thought?
2. Ask a question whose answer lives in a table, using words that appear in no cell. Do you get the table, or a paragraph about the table?
3. Ask "what's in this document?" If the answer is stitched from fragments, there is no document-scope retrieval.
4. Follow a citation. Can a sceptical reader land on the exact section it came from — heading path and all?
5. Re-ingest a changed document. Search for the old content. If it still answers, your index has ghosts.
6. Ask how tenants are separated. "We filter by tenant ID" and "tenants live in separate partitions" are very different answers.
7. Break it at volume. Ingest the whole corpus, not the sample folder. Rate limits, progress tracking, and failure recovery are part of accuracy — a half-ingested corpus answers confidently from half the truth.

The lesson

Prompt tweaks would have made v1's answers sound better. They could not have made the retrieved evidence better — because the evidence was damaged before the first question was ever asked, at the moment we decided a corpus of reports, tables, and code could all be sliced the same way.

At scale, accuracy isn't a query-time feature. It's an ingestion-time discipline:

• Respect the document's structure instead of overriding it.
• Address every chunk so answers can be verified, not just believed.
• Treat tables as data, and index what they mean, not only what they contain.
• Match retrieval units to question scope — sections for specifics, summaries for overviews.
• Let formats compete on relevance alone, from separate indexes.
• Isolate tenants at the storage layer, and make re-ingestion safe enough to be routine.

That's the work. It's less visible than a clever prompt — and it's why the client's team could finally put the system in front of users.

---

Building RAG over a corpus your users have to trust? This pipeline is part of our Applied AI practice — agent harnesses, RAG and knowledge systems, and fine-tuning, built on infrastructure you control, with permissions, grounded citations, and evals from the start. Scope an Applied AI engagement → — we'll reply within one business day.

FossibleWorks is a Healthcare IT and Applied AI implementation partner: self-hosted Medplum and FHIR, open-source HIS, AI agents, ERPNext, and DevOps — fossibleworks.com.