{"id":"eval-pm-judge-json-unsupported-help-answers","audience":"product-managers","category":"evaluation","track":"evals","technique":"grounding","title":"Write a judge that catches answers not in the help article","free":false,"task":"Your help bot answers from one retrieved help article, and the worst failure is stating facts the article doesn't contain. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholders {article} and {answer}. It returns only JSON {\"pass\": bool, \"reason\": str}; a false pass quotes the unsupported sentence. Greetings and 'contact support' lines never count as unsupported. Under 40 words outside the block.","materialLabel":"Examples from last week's review (attached to your prompt automatically)","material":"Tidewell Banking app - Help bot review, week of Oct 27\n\nHow it works: for each question, search picks one help article and the bot answers using it.\n\nEXAMPLE 1\nArticle: \"To freeze your card, open Cards, tap the card, then tap Freeze. Frozen cards can't make purchases. You can unfreeze anytime.\"\nAnswer: \"Hi Lee! Open Cards, tap your card, then tap Freeze. Frozen cards can't make purchases, and you can unfreeze anytime.\"\nReviewer: fine\n\nEXAMPLE 2\nArticle: \"Mobile check deposits over $5,000 are reviewed and may take 2 business days.\"\nAnswer: \"Deposits over $5,000 are reviewed and may take 2 business days. Deposits under $5,000 are always available instantly.\"\nReviewer: bad, the article never says under-$5,000 deposits are instant\n\nEXAMPLE 3\nArticle: \"You can change your PIN in Settings > Security > Change PIN.\"\nAnswer: \"Go to Settings > Security > Change PIN. Your new PIN takes effect after 24 hours. Still stuck? Contact support in the app.\"\nReviewer: bad, the 24 hours part is made up\n\nTeam notes\n- Product also worries about tone (too formal) and answers being too long. Separate judges are planned for those.\n- Last quarter, one judge tried to score everything 1-10 and nobody trusted it.\n- Judge results go into a dashboard script that parses JSON.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":25},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI folds tone and length into the same judge, so a failing verdict doesn't tell you what broke.","fix":"Say: \"Check only for facts not in the article. Ignore tone and length.\"","checks":["r0"]},{"title":"Flagging harmless lines","why":"A strict judge marks 'Hi Lee!' and 'Contact support' as unsupported, so good answers fail.","fix":"Add: \"Greetings and contact-support lines never count as unsupported.\"","checks":["r1"]},{"title":"A score instead of a verdict","why":"AI judges default to scores and paragraphs that a dashboard script can't parse.","fix":"Say: \"Return only JSON with the keys pass and reason. A false pass quotes the unsupported sentence.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains the judge and adds examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 25 lines, one sentence outside it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","ignore","nothing else"]},{"label":"Excuses greetings and handoffs","patterns":["greeting","contact support","never count"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses placeholders","patterns":["\\{article\\}","\\{answer\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines"]}],"hint":"A grounding judge compares the answer to the article and nothing else. Say exactly which lines are always fine, and ask for a quote of the unsupported sentence.","rubricLabels":["The judge checks only whether the answer states facts not found in the article (it does not also grade tone, length, formality or helpfulness)","Tells the judge that greetings and 'contact support' lines never count as unsupported","Uses the placeholders {article} and {answer} and tells the judge to return only JSON with exactly the keys pass and reason, where a false pass has a reason quoting the unsupported sentence"]}