Check if two PMs agree before you build an AI judge

About this exercise

Before building an AI judge, you and a teammate each labeled 10 outputs Pass or Fail, and you only trust the rubric if you agree at least 90% of the time. Write exactly 3 numbered lines: Disagreements (the ids, each with both labels), Agreement (the math, like '10 rows - 2 = 8 of 10 = 80%'), and Decision (Build the judge or Fix the rubric first, compared to 90%). Use any corrected labels. Plain text, under 70 words.

Common mistakes when you ask AI for this

• Missing the corrected label. The AI reads only the table, counts #7 as a disagreement, and gets 80%. Try: Add: "Use Rafa's corrected label for #7."
• Hiding the math. The AI says 'you mostly agree' without counts. Try: Ask: "Show the math, like 10 rows - 1 = 9 of 10 = 90%."
• No clear call. The AI says agreement is 'decent' instead of comparing to your bar. Try: Say: "End with Decision: Build the judge or Fix the rubric first, compared to 90%."
• A long analysis. The AI writes about labeling practices and rubric ideas. Try: Be exact: "3 numbered lines: Disagreements, Agreement, Decision."

What a strong prompt includes

• Uses the correction
• Asks for the math
• Gives the bar
• Says exactly how many
• Asks for plain text

Practice this now ↑

Related exercises

• Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
• Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
• Approve part of an overtime request with the math (Writing AI · Managers and team leads)
• Catch a chatbot's mistake on your invoice total (Data AI · Freelancers and creators)
• Catch a chatbot's mistakes about a supplier's return policy (Research & AI Search · Small business owners)