Write a JSON judge that catches personal data in summaries

About this exercise

Your incident-summary bot sometimes copies customer emails and phone numbers into summaries shared company-wide. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 25 lines, with the placeholder {summary}. It returns only JSON {"pass": bool, "reason": str}; a false pass quotes the leaked item. Internal employee IDs like E-1234 are allowed. Under 40 words outside the block.

Common mistakes when you ask AI for this

• One judge for everything. The AI adds timestamps, owners and tone to the same judge, so failures get mixed up. Try: Say: "Check only for customer emails and phone numbers."
• Flagging employee IDs. A strict judge treats E-2207 as personal data and fails good summaries. Try: Add: "Employee IDs like E-1234 are allowed."
• A score instead of a verdict. AI judges default to a risk score and an explanation, which the Slack bot can't parse. Try: Say: "Return only JSON with the keys pass and reason. A false pass quotes the leaked item."
• An essay around the prompt. The AI explains PII rules and adds sample outputs in extra code blocks. Try: Say: "Exactly 1 code block, at most 25 lines, one sentence after it."

What a strong prompt includes

• Limits it to one failure mode
• Allows employee IDs
• Asks for JSON pass/reason
• Uses a placeholder
• Limits the size

Practice this now ↑

Related exercises

• Add dark mode by overriding your existing CSS variables (Coding AI · Software developers)
• Add input checks to your number guessing game (Coding AI · High school students)
• Add labels to a signup form without rewriting the page (Coding AI · High school students)
• Announce the employee assistance program in three steps (Writing AI · HR and recruiting)
• Answer a patient portal message about delayed lab results (Writing AI · Nurses and healthcare workers)