{"id":"eval-cs-golden-answer-return-label-alaska","audience":"customer-support","category":"evaluation","track":"evals","technique":"context","title":"Write the golden answer for a return question with a new rule","free":false,"task":"You're writing test cases for your support bot, and this return question needs a golden answer. Since Nov 1, customers in Alaska and Hawaii get a prepaid label by mail instead of a printable one, arriving in 5-7 business days. Write exactly 3 numbered lines: '1. Answer: ...', '2. How: ...' and '3. Timing: ...'. Use only the help article plus that change. Plain text, under 60 words.","materialLabel":"The test question and the returns help article (attached to your prompt automatically)","material":"TEST CASE #12 for the Sundial Outfitters support bot\nCustomer (shipping address in Anchorage, Alaska): \"Hi, I want to return the hiking boots I got last week. Can I print a return label?\"\n\nHELP ARTICLE: \"How returns work\" (last updated in May)\n- You can return unworn items within 60 days of delivery.\n- Go to Orders > Return an item, choose the item, and print your prepaid label.\n- Drop the package at any post office.\n- Refunds go back to your original payment method within 5 business days after we receive the item.\n- Exchanges for a different size are free.\n\nTest-writing notes\n- The bot's answers in testing were long and often offered a 10% coupon, which it can't give.\n- Golden answers are the shortest complete correct answer we grade the bot against.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":60}],"mistakes":[{"title":"Using the old process","why":"The help article says to print a label, and the AI doesn't know Alaska changed on Nov 1.","fix":"Tell it: \"Since Nov 1, Alaska and Hawaii get a prepaid label by mail.\"","checks":["r0"]},{"title":"Leaving out the timing","why":"The AI says a label is coming but not when, so the customer writes back.","fix":"Ask: \"Timing: the label arrives in 5-7 business days.\"","checks":["r1"]},{"title":"Friendly extras","why":"The AI offers coupons or tips the bot can't actually give.","fix":"Say: \"Use only the article plus the change. No coupons.\"","checks":["r2"]},{"title":"A long reply","why":"The AI writes a full paragraph that's hard to grade against.","fix":"Show the shape: \"1. Answer: 2. How: 3. Timing:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives the new rule","patterns":["alaska","by mail","nov(ember)? 1"]},{"label":"Shows the three-line shape","patterns":["answer:","how:","timing:"]},{"label":"Includes the timing","patterns":["5-7|five to seven","business days"]},{"label":"Keeps to the article","patterns":["only.{0,30}(article|help)","no coupon"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer has to reflect today's rule even when the help article is behind. Give the AI the change, show the exact shape, and keep it to what the bot can actually offer.","rubricLabels":["Says the customer can return the boots but can't print a label, because Alaska addresses get a prepaid label by mail","The Timing line says the label arrives in 5-7 business days","The answer mentions no coupon or discount and nothing not in the help article or the Nov 1 change"]}