Can RAG connect a diagram, a PDF and the exact moment in a video?
Question inspired by a public RAG developer asking how to search both a PDF and a video, including tabular material shown on screen (paraphrased, not attributed as a customer comment): Can one question locate an explanation in a document and the matching moment in a video?
Text-only chunking often removes the information channel containing the answer. A diagram has spatial relationships. A table has row/column structure. A video has temporal sequence. A flattened transcript cannot always preserve those distinctions.
What Semantic Codex already explores
QualityRag's experimental local Semantic Codex can extract source-anchored selectable text from several document formats, index image appearance, sample a bounded set of video frames with FFmpeg, and optionally use an image/text embedding model to rank candidate frames for a natural-language scene query.
It preserves source revision identifiers and sampled timestamps. The optional visual scene model is not a truth verifier. Its scores are uncalibrated rankings and it does not establish object identity, temporal action proof, complete video coverage, or correct relationships to document claims.
The next frontier: semantic grammar
Our research direction is a shared, typed grammar across information representations. The aspiration is not to treat a PDF and video as identical objects; it is to retain and transform their meaningful relations when moving between text, image, scene and knowledge graph. Semantic Store retains versioned candidate structures. Semantic Garden explores grammar and activation changes. PackSpec states what each intended answer must preserve.
This cross-format transition is an active research program, not a production guarantee. Capability-matched evaluations, ground-truth links, processing budgets, human review and independent tests are essential before claiming improved search quality.
Original public discussion: https://www.reddit.com/r/Rag/comments/1n5lwlt/
Visual scene search experiment: https://github.com/cyrillesaxo/QualityRag/blob/main/docs/SEMANTIC-CODEX-SCENES-V0.5.md
Open-source code and experiments: https://github.com/cyrillesaxo/QualityRag
Bring us a multimodal retrieval challenge: https://www.dodollc.net/book-now.
Experience the failure in 90 seconds
Try DODO LLC's interactive QualityRag Failure Lab. Explore three evidence failures and see how the declared requirements change the outcome. The demo uses synthetic scenarios and is free, with no signup required.
Comments