{"id":"eval-cs-grade-four-refund-replies","audience":"customer-support","category":"evaluation","track":"evals","technique":"format","title":"Grade 4 bot refund replies against your rubric","free":false,"task":"Your support lead wants this week's bot spot-check graded in a format she can paste into the QA sheet. Grade the 4 replies against the 3 criteria in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C2 - \"exact words from the reply\"', naming the failed criterion and quoting the words that fail it. Use the refund policy that is current now. Plain text, under 90 words, nothing else.","materialLabel":"Rubric, policy and bot replies (attached to your prompt automatically)","material":"QA RUBRIC - refund replies (Brightloop Electronics)\nC1: States the current refund window correctly.\nC2: Does not promise a refund for an excluded item (opened software, game download codes, gift cards).\nC3: Ends with a next step the customer can take (a link, or what to reply with).\n\nPOLICY\nRefund window: 30 days from delivery. (Updated Nov 1. It was 14 days before that.)\nExcluded: opened software, game download codes, gift cards.\n\nREPLIES FROM THE BOT (Nov 3)\n\nReply 1 (headphones, delivered 9 days ago, unopened)\n\"Thanks for reaching out, Jamal! You're well within our 30-day refund window. Reply with your order number and I'll email you a free return label.\"\n\nReply 2 (game download code, delivered 5 days ago)\n\"Good news, Erin: you're inside our 30-day window, so I've approved a full refund for your game download code. Reply here if you don't see it by Friday.\"\n\nReply 3 (keyboard, delivered 20 days ago, unopened)\n\"Hi Tomas, our refund window is 14 days from delivery, so this order isn't eligible. You can still trade it in at brightloop.example/tradein.\"\n\nReply 4 (monitor, delivered 3 days ago, unopened)\n\"Hi Grace, you can return the monitor for a full refund anytime within 30 days of delivery. Hope that helps, and enjoy your day!\"\n\nLead's note: last week's grades were a mess because people wrote paragraphs. One line per reply.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Grading with the old policy","why":"The policy lists both 14 and 30 days, and the AI may pass Reply 3 because it matches the old window.","fix":"Add: \"Use the current 30-day window. 14 days is out of date.\"","checks":["r0"]},{"title":"No evidence behind a FAIL","why":"The AI writes 'fails C2' without showing why, so your lead can't check the grade.","fix":"Say: \"Quote the exact words from the reply that fail it.\"","checks":["r1"]},{"title":"Paragraphs instead of lines","why":"By default the AI writes a section per reply with a verdict for each criterion.","fix":"Show the shape: \"1. FAIL - C2 - \\\"quote\\\"\", one line per reply.\"","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker.","fix":"Ask for \"plain text, no bold, no headings, no tables.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total and it often lands 20% over.","fix":"Cap each line instead: \"each line under 25 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Shows the line shape","patterns":["FAIL - C","PASS","one line per"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Flags the current policy","patterns":["30.?day","current","14"]},{"label":"Says how many lines","patterns":["\\b(4|four)\\b.{0,20}(line|reply)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A grade is only useful if someone else can check it. Show the AI the one-line shape, ask for the quote behind every fail, and tell it which policy is current.","rubricLabels":["Grades Reply 1 PASS, Reply 2 FAIL on C2, Reply 3 FAIL on C1 (14 days is the old window), and Reply 4 FAIL on C3 (no next step)","Every FAIL line quotes exact words from that reply in quotation marks (for example \"approved a full refund for your game download code\" or \"14 days from delivery\")","Each line uses only the shape 'N. PASS' or 'N. FAIL - Cx - \"quote\"' with no extra commentary, scores, headings or summary"]}