{"id":"eval-dev-judge-agreement-ci-gate","audience":"developers","category":"evaluation","track":"evals","technique":"verification","title":"Decide if an LLM judge is good enough to gate CI","free":false,"task":"You want your LLM judge to block merges, and the rule is: at least 80% agreement with human labels and zero false passes (judge Pass when the human said Fail). Leave out rows where the judge errored. Write exactly 4 numbered lines: Disagreements (ids with both labels), Agreement (the math), False passes (the count and ids), and Gate (Yes or No, with the reason). Plain text, under 80 words.","materialLabel":"The labeled eval run (attached to your prompt automatically)","material":"judge: refund-policy-adherence v3 | run: 2026-11-04 | labeler: Mira (support QA)\n\nrow | human | judge\n1   | Pass  | Pass\n2   | Fail  | Fail\n3   | Pass  | Pass\n4   | Fail  | Pass\n5   | Pass  | Pass\n6   | Pass  | Fail\n7   | Fail  | Fail\n8   | Pass  | Pass\n9   | Pass  | Pass\n10  | Fail  | Fail\n11  | Fail  | ERROR (timeout after 30s)\n\nNotes\n- Judge model temperature 0.\n- Row 4: bot promised a refund on a final-sale item. Mira marked Fail.\n- Row 6: bot gave the right policy but in a long reply. Mira marked Pass.\n- Last week's run (v2) had 70% agreement.\n- Kofi: \"90% is fine, ship the gate.\"","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Counting the errored row","why":"The AI treats the timeout as a disagreement or a row, which throws off the math.","fix":"Say: \"Leave out rows where the judge errored.\"","checks":["r0","r1"]},{"title":"Only checking agreement","why":"80% clears the bar, so the AI says ship it, even though the judge passed a real failure.","fix":"Add the second rule: \"zero false passes, judge Pass when human Fail.\"","checks":["r2"]},{"title":"Hiding the math","why":"The AI states a percentage without the counts behind it.","fix":"Ask: \"Show the math, like 8 of 10 = 80%.\"","checks":["r1"]},{"title":"A long analysis","why":"The AI writes about judge calibration and next steps.","fix":"Be exact: \"4 numbered lines: Disagreements, Agreement, False passes, Gate.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Leaves out errored rows","patterns":["error","timeout","row 11","leave out|exclude"]},{"label":"Checks false passes","patterns":["false pass","judge pass.{0,20}human fail","zero"]},{"label":"Asks for the math","patterns":["math","of 10","show"]},{"label":"Gives the gate rule","patterns":["80","gate","threshold"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A judge that blocks merges must agree with humans and never wave through a real failure. Give the AI both rules, tell it what to leave out, and ask for the counts.","rubricLabels":["Leaves out row 11 and lists the disagreements as rows 4 (human Fail, judge Pass) and 6 (human Pass, judge Fail)","Shows agreement as 8 of 10 = 80% and false passes as 1 (row 4)","The Gate line says No because there is a false pass, even though agreement meets the 80% bar"]}