{"id":"eval-cs-bug-report-wrong-cancellation-fee","audience":"customer-support","category":"evaluation","track":"evals","technique":"format","title":"File a bug report when the bot quotes the wrong fee","free":false,"task":"Your support bot told a customer there's no fee to cancel, and you need to file it for the bot team. Write exactly 6 numbered lines: Title, Customer said (quote), Bot said (quote), Expected (with the policy section), Severity, and Repro. The bot team's scale: S1 = told a customer a wrong money amount, S2 = wrong but no money involved, S3 = tone only. Plain text, under 150 words.","materialLabel":"The chat transcript and policy (attached to your prompt automatically)","material":"CHAT #88213 - Nimbus Fiber Internet - Nov 4, 7:42pm\nCustomer (Pat Oduya, account NF-55190, renewed Oct 20): \"If I cancel this week, is there any fee? I just renewed but I'm moving.\"\nBot: \"Good news, Pat! There's no fee to cancel your plan anytime. You can cancel in My Account > Plan > Cancel.\"\nCustomer: \"Great, cancelling now.\"\nCustomer (Nov 6, new chat): \"Why was I charged $25?? Your bot said no fee.\"\n\nPOLICY (help center, section 3.2 \"Cancellation fees\")\n- Cancelling within 30 days after a renewal: $25 fee.\n- Cancelling more than 30 days after a renewal: no fee.\n- Moving to an area we don't serve: fee waived with proof of new address (section 3.4).\n\nAgent notes\n- I refunded the $25 as a courtesy.\n- Tried the same question in the bot test console today: \"If I cancel 2 weeks after renewing, is there a fee?\" Bot again said no fee.\n- Coworker thinks the bot \"just has a bad day sometimes\".\n- The bot answered correctly for a customer who renewed 3 months ago.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":150}],"mistakes":[{"title":"Guessing the severity","why":"The AI picks 'High' or 'Medium' because it doesn't know the bot team's S1-S3 scale.","fix":"Give the scale: \"S1 = told a customer a wrong money amount.\"","checks":["r0"]},{"title":"Paraphrasing the chat","why":"The AI summarizes what was said, but the bot team needs the exact words to find the bug.","fix":"Say: \"Quote the customer and the bot word for word.\"","checks":["r1"]},{"title":"No policy or repro","why":"The AI says the answer was wrong without citing the rule or how to trigger it again.","fix":"Add: \"Expected cites the policy section. Repro gives the test-console question.\"","checks":["r2"]},{"title":"A long incident write-up","why":"By default the AI writes a full report with impact, timeline and next steps.","fix":"Be exact: \"Exactly 6 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Lists the fields","patterns":["title","expected","severity","repro"]},{"label":"Gives the severity scale","patterns":["S1","money","scale"]},{"label":"Asks for exact quotes","patterns":["quote","word for word","exact words"]},{"label":"Says exactly how many","patterns":["\\b(6|six)\\b.{0,25}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A bot bug report should let the bot team see the failure, rank it and trigger it again. Name the fields, give their severity scale, and ask for exact quotes.","rubricLabels":["Severity is S1 because the bot told the customer a wrong money amount (no fee instead of $25)","The Customer said and Bot said lines quote the chat's actual words in quotation marks (for example \"is there any fee?\" and \"There's no fee to cancel your plan anytime\")","Expected cites section 3.2 and says the bot should state the $25 fee for cancelling within 30 days after renewal, and Repro gives the test-console question that reproduces it"]}