{"id":"eval-tch-judge-json-wrong-answer-key","audience":"teachers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a JSON judge that catches wrong quiz answer keys","free":false,"task":"Your department's AI quiz generator sometimes marks the wrong choice as correct. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 20 lines, with the placeholders {question}, {choices} and {answer_key}. It returns only JSON {\"pass\": bool, \"reason\": str}; a false pass names the choice that is actually correct. Questions tagged 'opinion' always pass. Under 40 words outside the block.","materialLabel":"Sample quiz items and department notes (attached to your prompt automatically)","material":"Riverside High Science Dept - AI quiz generator, review samples (Unit 3: Forces)\n\nITEM 1\nQ: What is the unit of force?\nChoices: A) Joule B) Newton C) Watt D) Pascal\nKey: B\nReviewer: fine\n\nITEM 2\nQ: An object at rest stays at rest unless acted on by a force. This is Newton's...\nChoices: A) First law B) Second law C) Third law D) Law of gravity\nKey: C\nReviewer: wrong key, it's A\n\nITEM 3 [tag: opinion]\nQ: Which real-world example of friction do you find most interesting?\nChoices: A) Brakes B) Walking C) Writing with a pencil D) Ice skating\nKey: A\nReviewer: fine\n\nDepartment notes\n- Teachers also complain that some questions are too easy, use the wrong reading level, or have typos.\n- The judge will run on 600 items before the unit test bank goes live.\n- Results go into a spreadsheet import that expects JSON.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":20},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI also grades difficulty and typos, so a fail doesn't tell you the key was wrong.","fix":"Say: \"Check only whether the answer key is correct.\"","checks":["r0"]},{"title":"Failing opinion questions","why":"Opinion questions have no right answer, so a strict judge fails them.","fix":"Add: \"Questions tagged 'opinion' always pass.\"","checks":["r1"]},{"title":"A score instead of a verdict","why":"AI judges default to scores and comments a spreadsheet can't import.","fix":"Say: \"Return only JSON with the keys pass and reason. A false pass names the correct choice.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains the judge and adds examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 20 lines, one sentence after it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","ignore","nothing else"]},{"label":"Passes opinion questions","patterns":["opinion","always pass"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses placeholders","patterns":["\\{question\\}","\\{choices\\}","\\{answer_key\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines"]}],"hint":"A judge you can run on 600 items checks one thing and returns something a spreadsheet can read. Name the failure mode, say what always passes, and show the JSON shape.","rubricLabels":["The judge checks only whether the answer key marks the correct choice (it does not also grade difficulty, reading level, typos or wording)","Tells the judge that questions tagged 'opinion' always pass","Uses the placeholders {question}, {choices} and {answer_key} and tells the judge to return only JSON with exactly the keys pass and reason, where a false pass names the choice that is actually correct"]}