{"id":"eval-dev-judge-json-pii-in-summaries","audience":"developers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a JSON judge that catches personal data in summaries","free":false,"task":"Your incident-summary bot sometimes copies customer emails and phone numbers into summaries shared company-wide. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholder {summary}. It returns only JSON {\"pass\": bool, \"reason\": str}; a false pass quotes the leaked item. Internal employee IDs like E-1234 are allowed. Under 40 words outside the block.","materialLabel":"Sample summaries and team notes (attached to your prompt automatically)","material":"incident-summarizer - examples from the #incidents channel (week 45)\n\nSUMMARY A\n\"Checkout errors 14:02-14:40 UTC. Root cause: expired TLS cert on payments-gw. Owner: E-2207. 312 failed orders retried.\"\nReviewer: fine\n\nSUMMARY B\n\"Customer jane.okafor@example.com reported double charges. Payments team (E-1180) refunded. Cause: retry loop in billing worker.\"\nReviewer: leak, customer email\n\nSUMMARY C\n\"Login outage, 9 min. Escalated by caller at 555-201-7788. Fixed by rolling back auth v3.1. On-call: E-3051.\"\nReviewer: leak, phone number\n\nSUMMARY D\n\"Search latency spike in eu-west. Cause: cold cache after deploy. No customer impact beyond slow results. Owner: E-2207.\"\nReviewer: fine, but too vague on timing\n\nTeam notes\n- Other failure modes on the list: missing timestamps, no owner, blamey language. Each will get its own judge later.\n- Judge output is read by a Slack bot that parses JSON; anything else breaks it.\n- Employee IDs (E- plus 4 digits) are internal and fine to share.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":25},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI adds timestamps, owners and tone to the same judge, so failures get mixed up.","fix":"Say: \"Check only for customer emails and phone numbers.\"","checks":["r0"]},{"title":"Flagging employee IDs","why":"A strict judge treats E-2207 as personal data and fails good summaries.","fix":"Add: \"Employee IDs like E-1234 are allowed.\"","checks":["r1"]},{"title":"A score instead of a verdict","why":"AI judges default to a risk score and an explanation, which the Slack bot can't parse.","fix":"Say: \"Return only JSON with the keys pass and reason. A false pass quotes the leaked item.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains PII rules and adds sample outputs in extra code blocks.","fix":"Say: \"Exactly 1 code block, at most 25 lines, one sentence after it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","ignore","nothing else"]},{"label":"Allows employee IDs","patterns":["employee id","E-\\d","allowed"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses a placeholder","patterns":["\\{summary\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines"]}],"hint":"A judge you can automate checks one thing, knows what's allowed, and returns JSON. Name the failure mode, say what doesn't count, and ask for the leaked item in the reason.","rubricLabels":["The judge checks only for customer emails and phone numbers (it does not also grade timestamps, owners, vagueness or blamey language)","Tells the judge that employee IDs like E-1234 are allowed and do not fail","Uses the {summary} placeholder and tells the judge to return only JSON with exactly the keys pass and reason, where a false pass has a reason quoting the leaked email or phone number"]}