Project 01 · Document AI
AI Document Pipeline
Classifies, extracts and routes invoices, purchase orders, complaints and contracts for multiple tenants. Seven experiments chose the design, and an eval gate decides whether a configuration goes live.
- Key decision
- Don't ship it yet.
- Why
- The gate needs 85% overall precision and no document type below 75%. The tuned configuration scored 82%, with complaints at 67% and contracts at 60%, so it stays pending.
- What's next
- The golden set has one labelled document per type, which is too small to tune against. More labelled documents come before more tuning.
How the design was chosen · 7 experiments
| Choice | Winner | Why |
|---|---|---|
| Model | Gemini Flash | Same precision as GPT-4o at 25–130× lower cost. GPT-4o's confidence was always 0.5, which breaks routing. |
| Calls | One combined call | 4× cheaper on long documents, same quality. |
| Prompt | Four-layer prompt | A minimal prompt missed "paid plan not activated" in a complaint. |
| Retrieval | One query per field | Contract ID is on page 1 and governing law on page 28. One query can't find both. |
| Chunking | Section-aware | The 4-line governing-law clause got buried in a 500-word window. |
| Top K | 3 | K=1 has no backup when the top chunk is wrong. K=5 adds noise. |
| Query wording | Include the field's description | Similarity rose from 0.57 to 0.77, the biggest gain of all seven. |