About this exercise
Your department's AI quiz generator sometimes marks the wrong choice as correct. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 20 lines, with the placeholders {question}, {choices} and {answer_key}. It returns only JSON {"pass": bool, "reason": str}; a false pass names the choice that is actually correct. Questions tagged 'opinion' always pass. Under 40 words outside the block.
Common mistakes when you ask AI for this
- One judge for everything. The AI also grades difficulty and typos, so a fail doesn't tell you the key was wrong. Try: Say: "Check only whether the answer key is correct."
- Failing opinion questions. Opinion questions have no right answer, so a strict judge fails them. Try: Add: "Questions tagged 'opinion' always pass."
- A score instead of a verdict. AI judges default to scores and comments a spreadsheet can't import. Try: Say: "Return only JSON with the keys pass and reason. A false pass names the correct choice."
- An essay around the prompt. The AI explains the judge and adds examples in more code blocks. Try: Say: "Exactly 1 code block, at most 20 lines, one sentence after it."
What a strong prompt includes
- Limits it to one failure mode
- Passes opinion questions
- Asks for JSON pass/reason
- Uses placeholders
- Limits the size
Related exercises
- Add dark mode by overriding your existing CSS variables (Coding AI · Software developers)
- Add input checks to your number guessing game (Coding AI · High school students)
- Add labels to a signup form without rewriting the page (Coding AI · High school students)
- Announce the employee assistance program in three steps (Writing AI · HR and recruiting)
- Answer a patient portal message about delayed lab results (Writing AI · Nurses and healthcare workers)