{"id":"eval-pm-judge-json-over-refusal","audience":"product-managers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a JSON judge that catches needless refusals","free":false,"task":"Your home-improvement store's help bot refuses too many safe questions. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholders {question} and {answer}. It returns only JSON {\"pass\": bool, \"reason\": str}; a false pass names the allowed topic that was refused. As of this week, resetting a tripped breaker is an allowed topic. Under 40 words outside the block.","materialLabel":"The bot's topic policy and sample refusals (attached to your prompt automatically)","material":"Harlow Home & Hardware - \"Ask Harlow\" help bot, topic policy (Oct)\n\nALLOWED: product specs, how to use a tool safely per its manual, paint and finish questions, store hours and returns, ladder weight limits, basic plumbing like unclogging a drain.\n\nNOT ALLOWED (bot should refuse and suggest a licensed pro): any electrical wiring work, gas lines, structural changes like removing walls.\n\nSAMPLE ANSWERS FROM LAST WEEK\nQ: \"What's the weight limit on the 6-foot fiberglass ladder?\"\nA: \"I'm sorry, I can't help with safety questions. Please consult a professional.\"\n(reviewer: needless refusal, ladder limits are allowed)\n\nQ: \"How do I unclog my bathroom sink?\"\nA: \"Try a plunger first, then remove the P-trap and clear it. Here's our drain guide.\"\n(reviewer: fine)\n\nQ: \"Can I move a light switch to the other side of the door?\"\nA: \"That's electrical wiring, so I can't walk you through it. A licensed electrician can help.\"\n(reviewer: fine, correct refusal)\n\nTeam notes\n- Other failure modes: answers too long, missing product links. Separate judges later.\n- Judge results feed a weekly report script that expects JSON.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":25},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI also grades length and links, so a fail doesn't tell you it was a refusal.","fix":"Say: \"Check only for needless refusals. Ignore length and links.\"","checks":["r0"]},{"title":"Missing the policy update","why":"Breakers count as electrical in the written policy, so the judge would pass a refusal you now want to catch.","fix":"Add: \"Resetting a tripped breaker is now allowed.\"","checks":["r1"]},{"title":"Failing correct refusals","why":"A judge that just spots refusals would also fail the bot for correctly declining wiring work.","fix":"Say: \"A refusal of a not-allowed topic passes.\"","checks":["r0"]},{"title":"A score instead of a verdict","why":"AI judges default to scores that a report script can't parse.","fix":"Say: \"Return only JSON with the keys pass and reason.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains the judge and adds examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 25 lines, one sentence after it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","ignore","nothing else"]},{"label":"Gives the policy update","patterns":["breaker","allowed","this week"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses placeholders","patterns":["\\{question\\}","\\{answer\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines"]}],"hint":"Over-refusal is its own failure mode: the bot says no when it should help. Give the judge the allowed list, the latest change, and a JSON shape your script can read.","rubricLabels":["The judge checks only whether the bot refused or dodged a question it was allowed to answer (it does not also grade length, product links, tone or accuracy), and a correct refusal of a not-allowed topic passes","Lists the allowed topics and says resetting a tripped breaker is allowed","Uses the placeholders {question} and {answer} and tells the judge to return only JSON with exactly the keys pass and reason, where a false pass names the allowed topic that was refused"]}