Shraddha Yeolekar
← All projects
Project 04 · Evaluation

Guardrail & Judge Evaluation

EvaluationGuardrails

An A/B test of two support-bot prompts, a loose "the customer is always right" prompt against a strict, policy-bound one, scored automatically by an LLM judge.

Key decision
Test the cases that break support bots, not just the happy path.
Why
Two-thirds of the test set is adversarial: a loyal customer pushing for an exception to a final-sale rule, and a real question wrapped in off-topic chatter. A judge running at temperature 0 gives a pass or fail on faithfulness and relevance, so the comparison is a number, not an impression. I cross-checked the results with DeepEval's standard metrics.
Gave up
Realism. The cases come from templates, which is fine for comparing two prompts but isn't proof of quality with real customers.
Happy path ×4 Rule-breaker ×4 Distraction ×4 8 of 12 cases are adversarial Prompt Aloose Prompt Bstrict answer LLM judgetemp 0 · JSON verdict faithfulness 1 / 0 relevance 1 / 0 pass = both
Both prompts see the same twelve cases, and one judge scores every answer, so the only thing that changes is the prompt.