Retrieval Is A Data Problem, Not A Vector Database Problem

When an AI assistant gives a wrong answer about a company’s own policies, the first instinct is to blame the retrieval layer. The chunks are too small. The embedding model is old. A different vector database would fix it. So the team spends two weeks re-indexing, tries a new chunking strategy, and the answers are still wrong in the same way.

The index was rarely the problem. In almost every case we look at, the answer was not in the documents, or it was in three places with two incompatible versions, or the person asking was not allowed to see the document that had it. No amount of similarity search fixes a corpus that does not contain the truth.

Retrieval quality is set before a single embedding is computed. Here is what to inspect first.

The demo corpus is not the corpus

Assistants are usually prototyped against a hand-picked set of documents: the clean PDF, the well-structured policy, the page someone rewrote last year. Those documents answer every question in the demo. They are also the best documents the company owns.

Production questions arrive against the whole estate — the 2019 handbook nobody archived, the spreadsheet a team keeps because the official system is slow, the policy that exists only as a paragraph in an email thread. The gap between the demo corpus and the real corpus is where most retrieval failures live, and it is measurable: take twenty real questions from the people who will use the thing, and count how many have a correct, current answer somewhere in the corpus you indexed. That number is your ceiling. Re-ranking cannot beat it.

Four questions per document, not per query

Before tuning anything, each source document has to pass four tests. They are cheap to run and they explain most bad answers.

Is it answerable? Some questions have no written answer because the knowledge lives in a person’s head or in how people actually behave. “What is the escalation path for a payment dispute?” may have an official answer and a real one. Retrieval will confidently return the official one.

Is it findable? A policy buried in a 200-page scanned PDF with no headings is technically in the corpus and practically invisible. Chunking cannot rescue content whose structure was never there. If a document needs a table of contents to be navigable by a human, the agent needs the same thing.

Is it current? Duplicate and stale versions are the single most common cause of confidently wrong answers. Two copies of a policy, one from 2022 and one from this year, both indexed, both retrievable, and the ranking decides which one the customer sees. Deduplication and an explicit freshness rule matter more than embedding choice.

Is it permitted? Access control is not a filter you add at the end. If the index contains HR files and the assistant serves the whole company, retrieval is now a data breach waiting for a well-worded question. This is the same scoping problem as deciding what an agent is allowed to touch, applied to reading rather than acting — and it has to be enforced at query time, from the identity of the person asking, not from the prompt.

Chunking is a judgement about your business

Generic chunk sizes are a default, not a decision. The right unit is the smallest piece of text that still means something on its own. A contract clause split mid-sentence is unusable. A benefit table split from its header row is worse than useless, because it looks like an answer.

Two practical rules cover most cases. Split on the document’s own structure first — headings, numbered clauses, table rows — and only fall back to fixed-size windows when there is no structure to follow. Then make sure every chunk carries its context: which document, which section, which version, which date. An answer the agent cannot attribute is an answer nobody can verify, and attribution is what makes a wrong answer debuggable instead of mysterious.

“I don’t know” is a feature

An assistant that answers every question will be wrong on some of them, and users cannot tell which. An assistant that says “I could not find this in the policy library — here are the two documents closest to it” is more useful, because it fails visibly and hands the question back to a person.

This has to be built, not hoped for. Set a relevance floor below which the agent does not answer. Give it a path for the unanswerable case — route to a human, log the question, and treat that log as your backlog of missing documents. The questions your assistant cannot answer are the clearest map of what your knowledge base is missing.

The same discipline applies to the review step. If retrieval returns a low-confidence match on something consequential, that is exactly where a human belongs in the workflow — not on every query, but on the ones the system already knows it is unsure about.

Test with real questions, then leave it alone

Build a fixed set of fifty to a hundred real questions with known correct answers, including at least ten that should return “not found.” Run it every time the corpus changes. This is the only way to know whether an index change helped or just moved the failures around, and it takes an afternoon to assemble from questions people already ask.

Then keep the corpus clean, which is ongoing work rather than a project. New documents have to be added, superseded ones removed, and the questions the assistant could not answer reviewed monthly. That maintenance is unglamorous, and it is the difference between an assistant that gets more useful over a year and one that quietly rots.

If you are planning a knowledge-grounded agent and want the corpus assessed before the index is built, that is the first thing we do in an engagement — see what we work on. The model is the easy part. The documents are the work.