Shraddha Yeolekar
← All projects
Project 05 · Retrieval

RAG Evaluation Framework

EvaluationRAG

An audit assistant for a dense rules document, built several ways to answer one question: when does retrieval beat handing a long-context model the whole document?

Key decision
Change only the retrieval, and hold everything else fixed.
Why
Every version uses the same model and the same auditor prompt, so any difference in the answers comes from what the model was given. Each version also logs its token count, so cost is measured alongside quality.
Design detail
A sub-section called "Exceptions" means little on its own. The last version embeds each chunk together with its full heading path, so the search knows which rules an exception belongs to.
VERSION WHAT THE MODEL OR SEARCH SEES Long context entire rules.md, with every question all tokens Heading chunks ### Exceptions | content… top 5 Heading + path PATH: Rules › Section › Exceptions | … top 5 same model, same prompt token count logged for every answer
Only the left side changes between versions. The highlighted heading path gives an isolated chunk the context of where it sits in the document.