{"id":"eval-pm-two-labeler-agreement-before-judge","audience":"product-managers","category":"evaluation","track":"evals","technique":"verification","title":"Check if two PMs agree before you build an AI judge","free":false,"task":"Before building an AI judge, you and a teammate each labeled 10 outputs Pass or Fail, and you only trust the rubric if you agree at least 90% of the time. Write exactly 3 numbered lines: Disagreements (the ids, each with both labels), Agreement (the math, like '10 rows - 2 = 8 of 10 = 80%'), and Decision (Build the judge or Fix the rubric first, compared to 90%). Use any corrected labels. Plain text, under 70 words.","materialLabel":"The two labelers' spreadsheet (attached to your prompt automatically)","material":"Pinehill Insurance - claims-letter assistant, labeling round 1\nRubric: Pass if the AI letter states the claim decision in the first sentence and gives the next step.\n\nid | you | Rafa\n1 | Pass | Pass\n2 | Pass | Pass\n3 | Fail | Fail\n4 | Pass | Fail\n5 | Fail | Fail\n6 | Pass | Pass\n7 | Fail | Pass\n8 | Pass | Pass\n9 | Fail | Fail\n10 | Pass | Pass\n\nNotes\n- Nov 5: Rafa re-read #7 and changed his label to Fail (the next step was missing).\n- #4: you counted \"Your claim is under review\" as a decision; Rafa didn't.\n- Labeling took about 25 minutes each.\n- The eng lead wants to start the judge on Monday.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Missing the corrected label","why":"The AI reads only the table, counts #7 as a disagreement, and gets 80%.","fix":"Add: \"Use Rafa's corrected label for #7.\"","checks":["r0","r1","r2"]},{"title":"Hiding the math","why":"The AI says 'you mostly agree' without counts.","fix":"Ask: \"Show the math, like 10 rows - 1 = 9 of 10 = 90%.\"","checks":["r1"]},{"title":"No clear call","why":"The AI says agreement is 'decent' instead of comparing to your bar.","fix":"Say: \"End with Decision: Build the judge or Fix the rubric first, compared to 90%.\"","checks":["r2"]},{"title":"A long analysis","why":"The AI writes about labeling practices and rubric ideas.","fix":"Be exact: \"3 numbered lines: Disagreements, Agreement, Decision.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Uses the correction","patterns":["correct","#7","changed"]},{"label":"Asks for the math","patterns":["math","of 10","show"]},{"label":"Gives the bar","patterns":["90","bar","threshold"]},{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"If two careful humans can't agree, an AI judge won't either. Ask for the disagreements, the math, and a call against your bar, using the latest labels.","rubricLabels":["Uses Rafa's corrected Fail for #7, so the only disagreement listed is #4 (you Pass, Rafa Fail)","Agreement shows the math as 10 rows - 1 = 9 of 10 = 90%","The Decision line says Build the judge because 90% meets the 90% bar"]}