Library › AI Evals › Customer support teams

Turn 3 live-traffic failures into test cases with expected answers

  • For Customer support teams
  • You practice Showing an example of what you want
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Three Pedalbot failures were found in live traffic. Turn each into one regression test case, based on the policy in the material, not on what the bot said. Output one JSON code block holding an array of exactly 3 objects with the keys id, type, user_message, expected, must_not. Use ids T1, T2, T3. Keep each rider's message exactly as written. Each expected is under 30 words and states what a correct reply does. Each must_not is under 12 words and names the wrong claim the bot made. No text outside the code block.

Common mistakes when you ask AI for this

  • Copying the bot's answer. The AI uses the bot's reply as the expected answer. Try: Say: "Expected comes from the policy, not from what the bot said."
  • Paraphrasing the rider. The AI translates or tidies the messages. Try: Add: "Keep each user_message exactly as written, including Spanish."
  • No must_not field. The AI writes only the right answer and skips what to forbid. Try: Add: "must_not names the wrong claim the bot made."
  • Prose around the JSON. The AI explains each case before and after the block. Try: Add: "One JSON code block and no text outside it."

What a strong prompt includes

  • Asks for JSON in one code block
  • Names the keys
  • Grounds expected in the policy
  • Keeps messages exact
  • Limits extra text

Practice this now ↑

Related exercises