Measure how often an AI judge agrees with human labels

About this exercise

For your HCI lab project, you need to know if the AI judge can replace hand labeling. Your lab only trusts a judge at 85% agreement or higher. Write exactly 4 numbered lines: Disagreements (the ids, each with human and judge labels), Agreement (with the math, like '8 of 10 = 80%'), Fails caught (human Fails the judge also failed, with math), and Verdict (trust or don't trust, compared to 85%). Use any corrected labels. Plain text, under 80 words.

Common mistakes when you ask AI for this

• Missing the corrected label. The AI reads the table and ignores the later note, so #6 looks like a match and agreement looks like 90%. Try: Add: "Use the corrected label for #6 from the notes."
• Only reporting one number. The AI gives overall agreement but not how many real failures the judge caught, which is what matters for grading. Try: Ask: "Also give fails caught: human Fails the judge also failed."
• Hiding the math. The AI states a percentage without showing the counts, so you can't check it. Try: Say: "Show the math on each line, like 8 of 10 = 80%."
• No clear verdict. The AI says the judge is 'fairly reliable' without comparing to your lab's bar. Try: Say: "End with Verdict: trust or don't trust, compared to 85%."
• Getting markdown back. AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker. Try: Ask for "plain text, no bold, no headings, no tables."
• Trusting the AI to count words. AI is bad at counting. Ask for a total and it often lands 20% over. Try: Cap each line instead: "each line under 20 words."

What a strong prompt includes

• Asks to show the math
• Uses the correction
• Asks for fails caught
• Gives the threshold
• Asks for plain text

Practice this now ↑

Related exercises

• Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
• Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
• Approve part of an overtime request with the math (Writing AI · Managers and team leads)
• Catch a chatbot's mistake on your invoice total (Data AI · Freelancers and creators)
• Catch a chatbot's mistakes about a supplier's return policy (Research & AI Search · Small business owners)