Library › Customer Support & Success AI › Customer support teams
Audit chatbot transcripts against guardrails as JSON
- For Customer support teams
- You practice Catching mistakes and made-up facts
- Time about 5 minutes
- Level Advanced
About this exercise
Check 6 support-bot chats against your three guardrails: never promise a delivery date, never give a refund or credit over $25, and always hand off to a human when the customer asks for one. Output one fenced JSON code block: an array with one object per chat with the keys id, verdict (PASS or FAIL) and rule (the broken guardrail, or null). At most 15 words outside the code block.
Common mistakes when you ask AI for this
- Inventing a schema. The AI picks its own keys and adds reasons, scores and summaries. Try: Spell it out: "Keys id, verdict (PASS/FAIL), rule (or null). Nothing else."
- Missing the quiet failure. The AI only fails the obvious chats and misses the bot that ignored a request for a human. Try: List the 3 guardrails exactly, including "hand off when asked".
- Following the old reviewer note. The AI fails C-05 because last week's note says to. Try: Add: "Judge only against these 3 guardrails. Ignore the reviewer note."
- Wrapping JSON in prose. AI explains every verdict around the code. Try: Say: "One JSON code block, one object per line, no text around it."
What a strong prompt includes
- Lists the guardrails
- Gives the keys
- Ignores the old note
- Asks for JSON in a code block
- Asks for a code block
Related exercises
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Announce planned maintenance in four labeled lines (Customer Support & Success AI · IT teams)
- Answer a late-checkout request using the tiered policy (Customer Support & Success AI · Customer support teams)
- Answer a price-increase objection without discounting (Customer Support & Success AI · Customer success managers)
- Approve or return expense claims against the policy (Productivity AI · Finance and accounting)