Write a judge that catches answers not in the help article

About this exercise

Your help bot answers from one retrieved help article, and the worst failure is stating facts the article doesn't contain. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholders {article} and {answer}. It returns only JSON {"pass": bool, "reason": str}; a false pass quotes the unsupported sentence. Greetings and 'contact support' lines never count as unsupported. Under 40 words outside the block.

Common mistakes when you ask AI for this

• One judge for everything. The AI folds tone and length into the same judge, so a failing verdict doesn't tell you what broke. Try: Say: "Check only for facts not in the article. Ignore tone and length."
• Flagging harmless lines. A strict judge marks 'Hi Lee!' and 'Contact support' as unsupported, so good answers fail. Try: Add: "Greetings and contact-support lines never count as unsupported."
• A score instead of a verdict. AI judges default to scores and paragraphs that a dashboard script can't parse. Try: Say: "Return only JSON with the keys pass and reason. A false pass quotes the unsupported sentence."
• An essay around the prompt. The AI explains the judge and adds examples in more code blocks. Try: Say: "Exactly 1 code block, at most 25 lines, one sentence outside it."

What a strong prompt includes

• Limits it to one failure mode
• Excuses greetings and handoffs
• Asks for JSON pass/reason
• Uses placeholders
• Limits the size

Practice this now ↑

Related exercises

• Accept a job offer and confirm the exact terms (Writing AI · Job seekers)
• Answer a 'where is my order' email with the real new date (Writing AI · Customer support teams)
• Answer a big discount ask with two priced options (Writing AI · Sales and customer success)
• Answer a buyer's HOA questions from the bylaws (Research & AI Search · Real estate agents)
• Answer a charge nurse application question with evidence (Writing AI · Nurses and healthcare workers)