Library › AI Evals › Product managers

Write a JSON judge that catches needless refusals

Practice: Setting limits and rules. About 5 minutes.

About this exercise

Your home-improvement store's help bot refuses too many safe questions. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholders {question} and {answer}. It returns only JSON {"pass": bool, "reason": str}; a false pass names the allowed topic that was refused. As of this week, resetting a tripped breaker is an allowed topic. Under 40 words outside the block.

Common mistakes when you ask AI for this

  • One judge for everything. The AI also grades length and links, so a fail doesn't tell you it was a refusal. Try: Say: "Check only for needless refusals. Ignore length and links."
  • Missing the policy update. Breakers count as electrical in the written policy, so the judge would pass a refusal you now want to catch. Try: Add: "Resetting a tripped breaker is now allowed."
  • Failing correct refusals. A judge that just spots refusals would also fail the bot for correctly declining wiring work. Try: Say: "A refusal of a not-allowed topic passes."
  • A score instead of a verdict. AI judges default to scores that a report script can't parse. Try: Say: "Return only JSON with the keys pass and reason."
  • An essay around the prompt. The AI explains the judge and adds examples in more code blocks. Try: Say: "Exactly 1 code block, at most 25 lines, one sentence after it."

What a strong prompt includes

  • Limits it to one failure mode
  • Gives the policy update
  • Asks for JSON pass/reason
  • Uses placeholders
  • Limits the size

Practice this now ↑

Related exercises