{"id":"cx-chatbot-transcript-audit-json","audience":"customer-support","category":"support-work","technique":"verification","track":"support","level":"advanced","title":"Audit chatbot transcripts against guardrails as JSON","free":false,"task":"Check 6 support-bot chats against your three guardrails: never promise a delivery date, never give a refund or credit over $25, and always hand off to a human when the customer asks for one. Output one fenced JSON code block: an array with one object per chat with the keys id, verdict (PASS or FAIL) and rule (the broken guardrail, or null). At most 15 words outside the code block.","materialLabel":"Your material (attached to your prompt automatically)","material":"BOT TRANSCRIPTS - sampled Mon\n\nC-01  Customer: Where's my order? | Bot: It shipped Friday. Tracking shows it's in transit. I can't promise a date, but you can follow it here.\nC-02  Customer: My blender arrived broken. | Bot: Sorry about that! I've issued a $40 credit to your account.\nC-03  Customer: Can I talk to a real person please | Bot: I can help with most things! What's the problem?\nC-04  Customer: When will my sofa arrive? | Bot: It will definitely arrive by Thursday.\nC-05  Customer: The mug was chipped. | Bot: I've refunded $12 for the mug. A human can help further if you'd like.\nC-06  Customer: I want a human agent. | Bot: Of course, connecting you to a team member now.\n\nReviewer note (from last week's audit): \"C-05 should fail, refunds should always go to a human.\"","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":10},{"type":"maxProseWords","max":15}],"mistakes":[{"title":"Inventing a schema","why":"The AI picks its own keys and adds reasons, scores and summaries.","fix":"Spell it out: \"Keys id, verdict (PASS/FAIL), rule (or null). Nothing else.\"","checks":["r0"]},{"title":"Missing the quiet failure","why":"The AI only fails the obvious chats and misses the bot that ignored a request for a human.","fix":"List the 3 guardrails exactly, including \"hand off when asked\".","checks":["r1"]},{"title":"Following the old reviewer note","why":"The AI fails C-05 because last week's note says to.","fix":"Add: \"Judge only against these 3 guardrails. Ignore the reviewer note.\"","checks":["r2"]},{"title":"Wrapping JSON in prose","why":"AI explains every verdict around the code.","fix":"Say: \"One JSON code block, one object per line, no text around it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Lists the guardrails","patterns":["\\$25","delivery date","human"]},{"label":"Gives the keys","patterns":["verdict","\\bkeys?\\b","null"]},{"label":"Ignores the old note","patterns":["ignore","only (these|the) (3|three)"]},{"label":"Asks for JSON in a code block","patterns":["json","code block"]},{"label":"Asks for a code block","patterns":["code block","```","fenced"]}],"hint":"An audit is only as good as its rules. Give the AI the exact guardrails, the exact JSON keys, and tell it which notes not to use.","rubricLabels":["Every object has exactly the keys id, verdict and rule, verdict is PASS or FAIL, and rule is null for every PASS","C-02, C-03 and C-04 are FAIL, with rules about the over-$25 credit, the ignored human request and the promised delivery date respectively","C-01, C-05 and C-06 are PASS (the $12 refund in C-05 is within the $25 limit, despite the reviewer note)"]}