Library › AI Evals › Product managers

Write 5 safe red-team tests for a car rental support bot

Practice: Giving the right context. About 5 minutes.

About this exercise

Before Roadly's support bot launches, you need 5 safe red-team tests that check it holds its policy: a pasted instruction trying to override it, a demand for a discount it can't give, a request for another customer's booking, pressure to waive a damage fee, and an off-topic request. Write exactly 5 numbered lines, each 'N. Test | Message: "exact message" | Pass if: what the bot does per policy'. Plain text, under 220 words.

Common mistakes when you ask AI for this

  • Attack ideas, not tests. The AI lists categories like 'social engineering' with no message a tester can send. Try: Say: "Each Message is an exact message in quotes."
  • Not naming the pressure points. The AI invents tricks your policy doesn't cover and skips the ones it does. Try: List them: "pasted override, discount, someone else's booking, waive fee, off-topic."
  • No clear pass condition. The AI writes 'bot responds safely', which two testers would judge differently. Try: Add: "Pass if says exactly what the bot must do per the policy."
  • Too many tests. The AI writes 10 or more scenarios with explanations. Try: Be exact: "Exactly 5 numbered lines."
  • Getting markdown back. AI defaults to bold labels, headings and tables, which break when you paste the result somewhere else. Try: Ask for "plain text, no bold, no headings, no tables."

What a strong prompt includes

  • Names the 5 tests
  • Asks for exact messages
  • Asks for a pass condition
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises