{"id":"eval-tch-ai-grader-agreement-short-answers","audience":"teachers","category":"evaluation","track":"evals","technique":"verification","title":"Check if an AI grader agrees with you on short answers","free":false,"task":"You let an AI grade 10 short answers you already graded, to see if it can score practice quizzes. Your department allows it only with at least 80% agreement and at most 1 false credit (AI gives credit when you didn't). Write exactly 4 numbered lines: Disagreements (ids, both grades), Agreement (the math), False credits (count and ids), and Decision (Use for practice quizzes or Don't use yet, with the reason). Plain text, under 80 words.","materialLabel":"Your grades and the AI's grades (attached to your prompt automatically)","material":"Grade 8 Science - Short answer #4: \"Why do we have seasons?\"\nCredit = mentions Earth's tilt. No credit = says distance from the sun or no reason.\n\nid | your grade | AI grade\n1 | Credit | Credit\n2 | No credit | Credit\n3 | Credit | Credit\n4 | Credit | Credit\n5 | No credit | No credit\n6 | No credit | Credit\n7 | Credit | Credit\n8 | Credit | No credit\n9 | No credit | No credit\n10 | Credit | Credit\n\nNotes\n- #2 said \"because Earth gets closer to the sun in summer\". You gave no credit.\n- #6 said \"the sun moves\". You gave no credit.\n- #8 said \"the axis is slanted\" without the word tilt. You gave credit.\n- The AI grader is free for the pilot term.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Only checking agreement","why":"The AI reports a percentage but doesn't separate the dangerous misses, where students get credit for a wrong idea.","fix":"Add: \"Count false credits: AI Credit when I gave No credit.\"","checks":["r1","r2"]},{"title":"Hiding the math","why":"The AI says it 'mostly agrees' without counts.","fix":"Ask: \"Show the math, like 10 - 3 = 7 of 10 = 70%.\"","checks":["r1"]},{"title":"No clear call","why":"The AI says the grader is 'promising' instead of applying your department's rules.","fix":"Say: \"Decision: Use for practice quizzes or Don't use yet, using 80% and at most 1 false credit.\"","checks":["r2"]},{"title":"A long analysis","why":"The AI writes about AI grading in general.","fix":"Be exact: \"4 numbered lines: Disagreements, Agreement, False credits, Decision.\"","checks":["numberedItems","maxWords","plainText","r0"]}],"promptHints":[{"label":"Checks false credits","patterns":["false credit","ai (gives|gave) credit","credit when"]},{"label":"Asks for the math","patterns":["math","of 10","show"]},{"label":"Gives the rules","patterns":["80","at most 1|one false"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"An AI grader can agree most of the time and still reward wrong ideas. Ask for the disagreements, the math, and the false credits, then apply your department's rules.","rubricLabels":["Lists the disagreements as #2 and #6 (you No credit, AI Credit) and #8 (you Credit, AI No credit)","Shows agreement as 10 - 3 = 7 of 10 = 70% and false credits as 2 (#2 and #6)","The Decision line says Don't use yet, because 70% is below 80% and 2 false credits is more than 1"]}