Decide if an LLM judge is good enough to gate CI

About this exercise

You want your LLM judge to block merges, and the rule is: at least 80% agreement with human labels and zero false passes (judge Pass when the human said Fail). Leave out rows where the judge errored. Write exactly 4 numbered lines: Disagreements (ids with both labels), Agreement (the math), False passes (the count and ids), and Gate (Yes or No, with the reason). Plain text, under 80 words.

Common mistakes when you ask AI for this

• Counting the errored row. The AI treats the timeout as a disagreement or a row, which throws off the math. Try: Say: "Leave out rows where the judge errored."
• Only checking agreement. 80% clears the bar, so the AI says ship it, even though the judge passed a real failure. Try: Add the second rule: "zero false passes, judge Pass when human Fail."
• Hiding the math. The AI states a percentage without the counts behind it. Try: Ask: "Show the math, like 8 of 10 = 80%."
• A long analysis. The AI writes about judge calibration and next steps. Try: Be exact: "4 numbered lines: Disagreements, Agreement, False passes, Gate."

What a strong prompt includes

• Leaves out errored rows
• Checks false passes
• Asks for the math
• Gives the gate rule
• Asks for plain text

Practice this now ↑

Related exercises

• Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
• Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
• Approve part of an overtime request with the math (Writing AI · Managers and team leads)
• Catch a chatbot's mistake on your invoice total (Data AI · Freelancers and creators)
• Catch a chatbot's mistakes about a supplier's return policy (Research & AI Search · Small business owners)