RAG Failure Lab: Three Evidence Failures You Can Test
Your RAG can retrieve the right document and still make the wrong decision. Try three short, hands-on evidence failures to see why.
Watch: QualityRag — The Evidence Engine
A narrated introduction to why relevant retrieval is not enough and how evidence requirements affect what an AI system should answer.
A 90-second experiment: would you trust the answer?
Scenario 1: The assistant cites an SOP, a mirrored wiki and a forwarded email. The three citations all point to the same origin. Does that satisfy a requirement for two independent sources?
Scenario 2: A warranty answer omits the applicable exclusion. A relevant coverage document is not evidence that the exception was checked.
Scenario 3: An old runbook is a strong similarity match, but its evidence falls outside the freshness limit.
What you see inside the lab
A short decision challenge: accept the AI answer, or check the governing requirements.
An evidence receipt showing source identity, declared origin, freshness, and the minimum evidence required.
A deterministic illustrative ADMIT or INCONCLUSIVE decision as the supplied evidence changes.
A reproducible technical path to the open-source QualityRag code.
What QualityRag does — and what it does not
PackSpec expresses the evidence-completeness rules. The base evaluator checks declared requirements using supplied evidence metadata. WitnessGraph and Semantic Codex research investigate relationships, source lineage, and representation across formats.
In this demo, evidence is synthetic and controlled. It is not a live assessment of your documents, and an ADMIT result does not prove source truth, real-world compliance, or authorization to act. Origin labels need reliable upstream ingestion; they are not independently authenticated by the demonstration.
Has this failure happened in your enterprise RAG?
DODO LLC offers a fixed-scope RAG Reliability Pilot to investigate an actual retrieval failure. The starting work defines representative failure cases, documents the governing evidence requirements, evaluates the observed behavior, and prepares a measurable remediation plan.
Current offer: $15,000 fixed-scope engagement. Scope, access, and deliverables are reviewed before engagement.
Frequently asked questions
Can I run this on my own confidential documents?
The free public demonstration uses synthetic cases and does not accept private customer documents. A scoped pilot can review appropriate data-handling controls before any customer evidence is shared.
Does QualityRag automatically prove a source is genuine?
No. The evaluator uses supplied origin and evidence metadata. Source authentication, document authority, and semantic equivalence require additional validated upstream checks.
Can I reproduce the results?
Yes. Explore the QualityRag repository and Academy examples. The published demo is an educational illustration of evidence requirements; verify each capability against the implementation and experiment notes.
Comments