Project 04 · Evaluation
Guardrail & Judge Evaluation
An A/B test of two support-bot prompts, a loose "the customer is always right" prompt against a strict, policy-bound one, scored automatically by an LLM judge.
- Key decision
- Test the cases that break support bots, not just the happy path.
- Why
- Two-thirds of the test set is adversarial: a loyal customer pushing for an exception to a final-sale rule, and a real question wrapped in off-topic chatter. A judge running at temperature 0 gives a pass or fail on faithfulness and relevance, so the comparison is a number, not an impression. I cross-checked the results with DeepEval's standard metrics.
- Gave up
- Realism. The cases come from templates, which is fine for comparing two prompts but isn't proof of quality with real customers.