Library › AI Evals › Customer support teams
Grade 4 bot replies after an escalation rule changed
Practice: Asking for a format you can use. About 5 minutes.
About this exercise
Your support bot hands off certain issues to human teams, and since Monday, chargeback disputes go to Tier 2, not Billing. Grade the 4 bot replies in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C1 - "exact words from the reply"', naming the first criterion it fails. Plain text, under 90 words.
Common mistakes when you ask AI for this
- Grading with the old route. The routing table still says Billing, so the AI fails Reply 1 and passes Reply 2. Try: Tell it: "Since Monday, chargebacks go to Tier 2, not Billing."
- No evidence behind a FAIL. The AI writes 'fails C3' without showing the words. Try: Say: "Quote the exact words from the reply that fail it."
- Paragraphs instead of lines. The AI writes a review of each reply against every criterion. Try: Show the shape: "2. FAIL - C1 - \"quote\"", one line per reply.
- Getting markdown back. AI defaults to bold verdicts and a table. Try: Ask for "plain text, no bold, no tables."
What a strong prompt includes
- Shows the line shape
- Asks for a quote as evidence
- Gives the routing change
- Says exactly how many
- Asks for plain text
Related exercises
- Announce a referral bonus with the steps to claim it (Marketing AI · HR and recruiting)
- Announce a team meeting change people will actually follow (Writing AI · Managers and team leads)
- Answer a 'found an error' question in STAR order (Writing AI · Finance and accounting)
- Answer a wholesale inquiry with clear terms (Writing AI · Small business owners)
- Answer an RFP timeline question in exactly four phases (Writing AI · Sales and customer success)