Project 05 · Retrieval
RAG Evaluation Framework
An audit assistant for a dense rules document, built several ways to answer one question: when does retrieval beat handing a long-context model the whole document?
- Key decision
- Change only the retrieval, and hold everything else fixed.
- Why
- Every version uses the same model and the same auditor prompt, so any difference in the answers comes from what the model was given. Each version also logs its token count, so cost is measured alongside quality.
- Design detail
- A sub-section called "Exceptions" means little on its own. The last version embeds each chunk together with its full heading path, so the search knows which rules an exception belongs to.