Library › AI Evals › Customer support teams
Turn "the bot feels pushy" into 4 testable checks
Practice: Catching mistakes and made-up facts. About 5 minutes.
About this exercise
Customers keep telling agents your cancellation bot 'feels pushy', and you need checks a QA reviewer can apply to one chat. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one chat] - e.g. "quote from the feedback"'. Don't use vague words like pushy, aggressive or salesy as the test, and skip praise and unrelated complaints. Plain text, under 130 words.
Common mistakes when you ask AI for this
- Keeping the vague words. The AI rewrites 'pushy' as 'respects the customer's choice', which is just as hard to grade. Try: Say: "Each check must be something you can see or count in one chat."
- No example from real data. The AI writes generic retention tips instead of tying checks to what customers said. Try: Add: "Quote an example from the feedback on each line."
- Banning all offers. The AI fails any chat with an offer, but retention wants to keep one. Try: Add: "One save offer is fine. Skip praise and unrelated complaints."
- Too many checks. The AI turns every comment into a check. Try: Be exact: "Exactly 4 numbered checks."
What a strong prompt includes
- Asks for observable checks
- Asks for quoted examples
- Says exactly how many
- Allows one offer
- Asks for plain text
Related exercises
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
- Approve part of an overtime request with the math (Writing AI · Managers and team leads)
- Brief a research agent to prep you for a sales call (Agents & Automation AI · Sales and customer success)
- Brief a research assistant to shortlist sources you can trust (Agents & Automation AI · College students)