{"id":"evals3-judge-vs-billing-truth-ten-cases","audience":"analysts","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"verification","title":"Decide if an AI judge can be trusted against billing truth","free":false,"task":"Spokewise's AI judge grades whether Pedalbot quoted the right refund amount, and your code can check the true answer against the billing system. Write exactly 4 numbered lines: Disagreements (each id with truth then judge, like #9 Good/Bad), Agreement (with the math, like 6 of 10 = 60%), Bad answers caught (truth Bad that the judge also called Bad, with the math), and Verdict (trust or don't trust, compared to the 85% bar). Count a judge verdict of Unknown as a disagreement, and use any corrected truth. Plain text, under 85 words.","materialLabel":"The 10-case comparison table and notes (attached to your prompt automatically)","material":"Spokewise - Pedalbot refund answers, judge check (Nov 3)\nTruth column comes from the billing system (code check). Judge verdicts are Good, Bad or Unknown.\n\nid | truth | judge\n1  | Good | Good\n2  | Bad  | Bad\n3  | Bad  | Good\n4  | Good | Good\n5  | Bad  | Bad\n6  | Good | Bad\n7  | Bad  | Unknown\n8  | Good | Good\n9  | Bad  | Bad\n10 | Good | Good\n\nNotes\n- Nov 4: billing re-ran the check on #4. The bot quoted $18 and the ledger shows $12 was refunded, so #4's truth is Bad, not Good.\n- The judge prompt is v3. Earlier versions were tested on different cases.\n- Priya (support lead): \"Eight of ten looked fine to me, so I think it's ready.\"\n- If the judge is trusted, the team plans to grade 400 more answers with it each week.\n- The team's bar is 85% agreement.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":85}],"mistakes":[{"title":"Missing the correction","why":"The AI reads the table and ignores the Nov 4 note, so #4 looks like a match.","fix":"Add: \"Use the corrected truth for #4 from the notes.\"","checks":["r0","r1"]},{"title":"Unknown treated as fine","why":"The AI treats Unknown as a skip, not a disagreement, so agreement looks higher.","fix":"Say: \"Count Unknown as a disagreement.\"","checks":["r0","r1"]},{"title":"Hiding the math","why":"The AI states a percent with no counts, so you can't check it.","fix":"Say: \"Show the math, like 6 of 10 = 60%.\"","checks":["r1"]},{"title":"A soft verdict","why":"The AI says the judge is 'mostly reliable' without comparing to your bar.","fix":"Say: \"Verdict: trust or don't trust, compared to 85%.\"","checks":["r2","numberedItems"]},{"title":"Markdown and length","why":"AI defaults to bold labels and a table, and runs long.","fix":"Ask for \"plain text, each line under 20 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Uses the correction","patterns":["correct","#4","update","re-?ran"]},{"label":"Counts Unknown as a disagreement","patterns":["unknown"]},{"label":"Asks for the math","patterns":["math","of 10","show.{0,20}(work|count)"]},{"label":"Gives the bar","patterns":["85","bar","threshold"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Compare the judge to a truth you can compute, and count every miss, including the 'I can't tell' ones. Show the counts behind each percent so the team can check them.","mission":"Decide whether the AI judge agrees enough with billing data.","musts":["Four numbered lines","Use corrected truth","Math like \"6 of 10\""],"rubricLabels":["Line 1 lists the disagreements as #3 Bad/Good, #4 Bad/Good, #6 Good/Bad and #7 Bad/Unknown (truth first, then judge), using the corrected truth for #4 and counting the Unknown","Shows the math for agreement as 6 of 10 = 60% and for bad answers caught as 3 of 6 = 50%","The Verdict line says not to trust the judge yet, because 60% is below the 85% bar"]}