My RAG works with one PDF. Why does it fail when I add more documents?
Question raised by a RAG developer in a public community (paraphrased; no customer identity claimed): The chatbot is accurate with one PDF, but returns irrelevant answers after adding more PDFs. Why?
More documents mean more opportunities for near-duplicate passages, stale versions, conflicting definitions, different document permissions, and answers spread across several sources. Increasing top-k can add noise without restoring missing relationships.
What to investigate
Inspect actual candidate retrieval sets before generation. Track source versions and permissions. Compare dense retrieval with keyword and metadata filters. Evaluate questions requiring evidence from two or more documents, and keep exact-match identifiers and boundary conditions. Ask not only whether a relevant passage appeared, but whether the full required set of evidence was present.
What is different about QualityRag?
QualityRag connects evidence completeness contracts (PackSpec) with experimental Semantic Store and typed relation traversal. WitnessGraph-style evidence requirements ask which source-linked witnesses must exist for a particular claim. The current four-regime comparison demonstrates progressively richer candidate operations, but does not claim independently validated source truth or superior production accuracy.
From better chunks to better information structures
Our longer-term research is about grammar-induced information representation: agents propose candidate relations and transformations, evaluate preservation obligations, and retain source-bound lineage. A good representation is the one that can support the questions and constraints that matter, not the one with the most vectors.
Original public question: https://www.reddit.com/r/Rag/comments/1ne4lbi/
Research demonstration and limitations: https://github.com/cyrillesaxo/QualityRag/blob/main/docs/SEMANTIC-CODEX-FOUR-REGIMES-V1.md
Open-source repository: https://github.com/cyrillesaxo/QualityRag
Have a repeatable multi-document retrieval failure? Request a paid pilot: https://www.dodollc.net/book-now.
Experience the failure in 90 seconds
Try DODO LLC's interactive QualityRag Failure Lab. Explore three evidence failures and see how the declared requirements change the outcome. The demo uses synthetic scenarios and is free, with no signup required.
Comments