Shraddha Yeolekar
← All projects
Project 01 · Document AI

AI Document Pipeline

EvaluationRAG

Classifies, extracts and routes invoices, purchase orders, complaints and contracts for multiple tenants. Seven experiments chose the design, and an eval gate decides whether a configuration goes live.

Key decision
Don't ship it yet.
Why
The gate needs 85% overall precision and no document type below 75%. The tuned configuration scored 82%, with complaints at 67% and contracts at 60%, so it stays pending.
What's next
The golden set has one labelled document per type, which is too small to tune against. More labelled documents come before more tuning.
Uploaddocument Longdocument? no: whole document Gemini Flashclassify + extract confidence≥ bar? auto-approve human review long docs yes Section-awarechunks Query per fieldtop 3 each per-field context every change Configchange Eval gategolden set pass Live configprompt · chunks · K sets fail → stays pending
How a document moves through the pipeline (top two lanes) and how a change reaches production (bottom lane). Nothing in the live configuration changes without passing the gate.
How the design was chosen · 7 experiments
ChoiceWinnerWhy
ModelGemini FlashSame precision as GPT-4o at 25–130× lower cost. GPT-4o's confidence was always 0.5, which breaks routing.
CallsOne combined call4× cheaper on long documents, same quality.
PromptFour-layer promptA minimal prompt missed "paid plan not activated" in a complaint.
RetrievalOne query per fieldContract ID is on page 1 and governing law on page 28. One query can't find both.
ChunkingSection-awareThe 4-line governing-law clause got buried in a 500-word window.
Top K3K=1 has no backup when the top chunk is wrong. K=5 adds noise.
Query wordingInclude the field's descriptionSimilarity rose from 0.57 to 0.77, the biggest gain of all seven.
0% 100% 100% Invoice 100% PO 67% Complaint 60% Contract floor 75% Overall 82% ship bar 85% HELD
The gate judges the whole system, not its best cases. With one document per type, read these as a method, not a benchmark.