{"id":"evals3-judge-saw-the-answer-key","audience":"product-managers","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"verification","title":"Spot why a judge scored 10 of 10 and tell your team","free":false,"task":"Omar wants to switch Pedalbot's grading to the judge from the trial with the best score. Reply to Omar in a plain text Slack message of at most 4 sentences that starts with \"Omar,\". Name the trial with the setup flaw and the flaw, say why its score does not measure the judge, give the honest agreement score as a fraction and a percent, and end by saying what to rerun. Under 90 words.","materialLabel":"The three judge trials (attached to your prompt automatically)","material":"Spokewise - judge trials on the same 10 Pedalbot replies (human labels are the reference)\nEach trial counted how often the AI judge's Pass/Fail matched the human label.\n\nTrial 1\nJudge prompt inputs: {policy}, {customer_question}, {reply}\nAgreement with humans: 7 of 10\n\nTrial 2\nJudge prompt inputs: {policy}, {customer_question}, {reply}, {human_label}\nJudge instruction: \"Decide Pass or Fail. A human reviewer already marked this reply.\"\nAgreement with humans: 10 of 10\n\nTrial 3\nJudge prompt inputs: {customer_question}, {reply}\nAgreement with humans: 5 of 10\n\nNotes\n- Same judge model and same 10 replies in all three trials.\n- Omar: \"Trial 2 is clearly the best. Let's ship that judge and stop labeling by hand.\"\n- Trial 3 left out the policy to save tokens.\n- The team will grade about 600 replies a week once a judge is chosen.\n- Humans needed about 50 minutes to label the 10 replies.","rules":[{"type":"maxSentences","max":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Picking the best score","why":"The AI agrees Trial 2 is best because 10 of 10 is the highest number.","fix":"Say: \"Check what each trial's judge was shown before trusting its score.\"","checks":["r0"]},{"title":"Reporting the wrong number","why":"The AI repeats 10 of 10 or averages the trials.","fix":"Say: \"Give the honest score: the fair trial with the policy, as a fraction and a percent.\"","checks":["r1"]},{"title":"No next step","why":"The AI explains the flaw but doesn't say what to rerun.","fix":"Add: \"End by saying what to rerun.\"","checks":["r2"]},{"title":"Too long and formatted","why":"AI defaults to bold, bullets and a long explanation.","fix":"Ask for \"plain text, at most 4 sentences, under 90 words.\"","checks":["maxSentences","plainText","maxWords"]}],"promptHints":[{"label":"Asks to check what the judge saw","patterns":["what.{0,30}(saw|shown|given)","inputs?","human_?label","answer key"]},{"label":"Asks for fraction and percent","patterns":["fraction","percent","%","of 10"]},{"label":"Names the reader","patterns":["omar"]},{"label":"Asks for a next step","patterns":["rerun","next step","end (by|with)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A judge that is shown the answer is just copying it. Look at the inputs of each trial, find the one that includes the label, and trust only the fair trial.","mission":"Explain why the perfect judge score cannot be trusted.","musts":["Name the flawed trial","Give the honest number","Four sentences, start Omar,"],"rubricLabels":["Names Trial 2 as the flawed trial because the judge was shown the human label (the answer key), so its 10 of 10 does not measure how well the judge works","Gives Trial 1's 7 of 10 = 70% as the honest agreement score, and does not recommend the Trial 2 setup","Starts with \"Omar,\" and ends by saying to rerun without the human label (or to keep humans labeling until a fair trial passes)"]}