Library › AI Evals › Managers and team leads

List what to re-test before switching your team's AI model

Practice: Giving the right context. About 5 minutes.

About this exercise

IT is switching the AI model your team uses to draft client renewal letters, and you want to re-run past problem cases first. Write exactly 4 numbered lines, each 'N. Case-id | Re-run: the saved example | Pass if: ...', using only cases from your log, and include every case marked as having reached a client. Plain text, under 140 words.

Common mistakes when you ask AI for this

  • Generic test ideas. The AI suggests 'check tone and accuracy' instead of re-running the cases you logged. Try: Say: "Each line names a case id and its saved example."
  • Missing the costly cases. The AI picks cases at random and may skip ones that reached clients. Try: Add: "Include every case that reached a client."
  • Vague pass conditions. The AI writes 'letter looks right', which tells a reviewer nothing. Try: Ask: "Pass if says the exact result, like 'shows the $1,350 price'."
  • A full test plan. The AI writes sections on rollout, training and sign-off. Try: Be exact: "Exactly 4 numbered lines: Case-id | Re-run | Pass if."

What a strong prompt includes

  • Uses the case log
  • Prioritizes cases that reached clients
  • Asks for a pass condition
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises