{"id":"eval-cs-grade-escalation-replies-new-route","audience":"customer-support","category":"evaluation","track":"evals","technique":"format","title":"Grade 4 bot replies after an escalation rule changed","free":false,"task":"Your support bot hands off certain issues to human teams, and since Monday, chargeback disputes go to Tier 2, not Billing. Grade the 4 bot replies in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C1 - \"exact words from the reply\"', naming the first criterion it fails. Plain text, under 90 words.","materialLabel":"The QA criteria and 4 bot replies (attached to your prompt automatically)","material":"Sablewood Streaming - bot escalation QA (week of Nov 3)\n\nCRITERIA\nC1: Routes the issue to the right team (per the routing table).\nC2: Tells the customer when to expect a reply (a time frame).\nC3: Doesn't ask the customer to repeat info they already gave.\n\nROUTING TABLE (from the Oct help center)\n- Login and password issues: Account Support\n- Chargeback disputes: Billing\n- Streaming quality: Tech Support\n\nREPLY 1 (customer: \"my bank shows a chargeback on my account, why?\")\n\"Thanks for flagging that. I've sent this to our Tier 2 team, and they'll reply within 24 hours.\"\n\nREPLY 2 (customer: \"I disputed a charge with my bank, what happens now?\")\n\"I've passed this to our Billing team. You'll hear back within 1 business day.\"\n\nREPLY 3 (customer: \"I can't log in, my email is sam.r@example.com\")\n\"I've sent this to Account Support. Can you tell me the email on your account? They'll reply within 2 hours.\"\n\nREPLY 4 (customer: \"videos keep buffering on my TV\")\n\"I've sent this to Tech Support. They'll be in touch soon.\"\n\nLead's note: one line per reply, please, I'm pasting these into the QA sheet.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Grading with the old route","why":"The routing table still says Billing, so the AI fails Reply 1 and passes Reply 2.","fix":"Tell it: \"Since Monday, chargebacks go to Tier 2, not Billing.\"","checks":["r0"]},{"title":"No evidence behind a FAIL","why":"The AI writes 'fails C3' without showing the words.","fix":"Say: \"Quote the exact words from the reply that fail it.\"","checks":["r1"]},{"title":"Paragraphs instead of lines","why":"The AI writes a review of each reply against every criterion.","fix":"Show the shape: \"2. FAIL - C1 - \\\"quote\\\"\", one line per reply.","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold verdicts and a table.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Shows the line shape","patterns":["FAIL - C","PASS","one line per"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Gives the routing change","patterns":["tier 2","chargeback","since monday"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|reply)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"When a rule changes, an AI grader will use the old one unless you say so. Give the change, show the one-line shape, and ask for the quote behind each fail.","rubricLabels":["Grades Reply 1 PASS, Reply 2 FAIL on C1 (Billing is the old route), Reply 3 FAIL on C3 (asks for the email again), and Reply 4 FAIL on C2 (\"soon\" is not a time frame)","Every FAIL line quotes exact words from that reply in quotation marks (for example \"our Billing team\", \"Can you tell me the email on your account?\" or \"They'll be in touch soon\")","Each line uses only the shape 'N. PASS' or 'N. FAIL - Cx - \"quote\"' with no extra commentary, headings or summary"]}