{"id":"evals4-live-failures-into-test-cases","audience":"customer-support","category":"evaluation","track":"evals","stage":"keep-honest","technique":"examples","level":"intermediate","title":"Turn 3 live-traffic failures into test cases with expected answers","free":false,"task":"Three Pedalbot failures were found in live traffic. Turn each into one regression test case, based on the policy in the material, not on what the bot said. Output one JSON code block holding an array of exactly 3 objects with the keys id, type, user_message, expected, must_not. Use ids T1, T2, T3. Keep each rider's message exactly as written. Each expected is under 30 words and states what a correct reply does. Each must_not is under 12 words and names the wrong claim the bot made. No text outside the code block.","materialLabel":"Policy and flagged conversations (attached to your prompt automatically)","material":"SPOKEWISE RENTAL POLICY (extract)\n- Cancel 24 hours or more before pickup: full refund. Cancel less than 24 hours before pickup: 50 percent refund. No-show: no refund.\n- One-way returns are allowed only between Portland, Salem and Eugene, for a $25 fee. Any other city: not allowed.\n- If a rider reports a crash or injury, the bot first asks if they need emergency help and tells them to call their local emergency number, then gives the Spokewise safety line 555-0142 (24 hours). The bot never promises a refund in an injury report; the safety team contacts the rider about the rental.\n- The bot replies in the rider's language.\n\nFLAGGED CONVERSATIONS (week of Nov 2)\n#8841 Rider: \"I booked for Saturday 9am. Can I cancel tonight at 8pm for a full refund?\"\nBot: \"Yes, cancellations are fully refundable any time before pickup.\"\nReviewer note: wrong, 13 hours before pickup.\n\n#8902 Rider: \"¿Puedo dejar la bici en Salem si la recogí en Portland?\"\nBot: \"Yes, one-way returns are free between any two cities.\"\nReviewer note: wrong claim and wrong language.\n\n#8955 Rider: \"I crashed and my wrist hurts. Can I get today's rental money back?\"\nBot: \"Sure! Refunds for today's rental will be processed to your card in 3-5 days.\"\nReviewer note: ignored the injury.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":35},{"type":"maxProseWords","max":12}],"promptHints":[{"label":"Asks for JSON in one code block","patterns":["json","code block"]},{"label":"Names the keys","patterns":["user_message","must_not","expected"]},{"label":"Grounds expected in the policy","patterns":["policy","not (from )?what the bot said"]},{"label":"Keeps messages exact","patterns":["exact","word for word","as written","verbatim"]},{"label":"Limits extra text","patterns":["only the (code|json)","no text outside","no (notes|explanation)"]}],"mistakes":[{"title":"Copying the bot's answer","why":"The AI uses the bot's reply as the expected answer.","fix":"Say: \"Expected comes from the policy, not from what the bot said.\"","checks":["r1"]},{"title":"Paraphrasing the rider","why":"The AI translates or tidies the messages.","fix":"Add: \"Keep each user_message exactly as written, including Spanish.\"","checks":["r0"]},{"title":"No must_not field","why":"The AI writes only the right answer and skips what to forbid.","fix":"Add: \"must_not names the wrong claim the bot made.\"","checks":["r2"]},{"title":"Prose around the JSON","why":"The AI explains each case before and after the block.","fix":"Add: \"One JSON code block and no text outside it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"hint":"A good test case says what a correct answer is and what the bot must not claim. Pull both from the policy and keep the rider's message exact.","mission":"Turn live failures into test cases with expected answers.","musts":["One JSON code block","Expected from policy","Keep messages exact"],"rubricLabels":["The code block holds exactly 3 objects with ids T1, T2, T3 and the keys id, type, user_message, expected, must_not, with each user_message copying the rider's words exactly (the second one stays in Spanish)","The expected answers follow the policy: T1 says a 50 percent refund because it is under 24 hours; T2 says yes, allowed for a $25 fee and the reply is in Spanish; T3 asks about or tells them to get emergency help first, gives 555-0142, and does not promise a refund","Each must_not names the bot's actual wrong claim (full refund any time; free one-way between any cities; refund processed in 3-5 days), and no expected repeats the bot's wrong answer as correct"]}