{"id":"eval-col-judge-human-agreement-rate","audience":"college","category":"evaluation","track":"evals","technique":"verification","title":"Measure how often an AI judge agrees with human labels","free":false,"task":"For your HCI lab project, you need to know if the AI judge can replace hand labeling. Your lab only trusts a judge at 85% agreement or higher. Write exactly 4 numbered lines: Disagreements (the ids, each with human and judge labels), Agreement (with the math, like '8 of 10 = 80%'), Fails caught (human Fails the judge also failed, with math), and Verdict (trust or don't trust, compared to 85%). Use any corrected labels. Plain text, under 80 words.","materialLabel":"Your labeling spreadsheet export (attached to your prompt automatically)","material":"HCI 340 - Study Buddy chatbot eval - labels export (Nov 4)\nTask: does the chatbot's answer cite the course reading correctly? Pass/Fail.\n\nid | human label (Kai) | AI judge label\n1  | Pass | Pass\n2  | Fail | Fail\n3  | Fail | Pass\n4  | Pass | Pass\n5  | Pass | Pass\n6  | Pass | Pass\n7  | Pass | Pass\n8  | Fail | Fail\n9  | Pass | Pass\n10 | Fail | Fail\n\nNotes:\n- Nov 5 update from Dr. Okafor: after a second look, #6 is a Fail (it cited the wrong chapter). Kai's original Pass was a mistake.\n- The judge prompt is v2. v1 had 70% agreement on a different set of 20 rows.\n- Lab meeting Thursday. Lena wants to label 500 more rows with the judge if it's good enough.\n- Kai labeled these in about 40 minutes.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Missing the corrected label","why":"The AI reads the table and ignores the later note, so #6 looks like a match and agreement looks like 90%.","fix":"Add: \"Use the corrected label for #6 from the notes.\"","checks":["r0","r2"]},{"title":"Only reporting one number","why":"The AI gives overall agreement but not how many real failures the judge caught, which is what matters for grading.","fix":"Ask: \"Also give fails caught: human Fails the judge also failed.\"","checks":["r1"]},{"title":"Hiding the math","why":"The AI states a percentage without showing the counts, so you can't check it.","fix":"Say: \"Show the math on each line, like 8 of 10 = 80%.\"","checks":["r0","r1"]},{"title":"No clear verdict","why":"The AI says the judge is 'fairly reliable' without comparing to your lab's bar.","fix":"Say: \"End with Verdict: trust or don't trust, compared to 85%.\"","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker.","fix":"Ask for \"plain text, no bold, no headings, no tables.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total and it often lands 20% over.","fix":"Cap each line instead: \"each line under 20 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Asks to show the math","patterns":["math","of 10","show.{0,20}(work|count|calc)","x of y"]},{"label":"Uses the correction","patterns":["correct","#6","update"]},{"label":"Asks for fails caught","patterns":["fails? caught","caught","recall","human fail"]},{"label":"Gives the threshold","patterns":["85","threshold","bar"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"An AI judge is only worth using if it agrees with careful humans. Ask for the counts behind every percentage, check the real failures separately, and compare against your bar.","rubricLabels":["Uses the corrected label for #6 (Fail) and reports agreement as 8 of 10 = 80%, showing the math (listing the matching ids is fine)","Lists the disagreements as #3 and #6 (human Fail, judge Pass for both) and reports fails caught as 3 of 5 = 60%, showing the math","The Verdict line says not to trust the judge yet because 80% is below the 85% bar"]}