Library › AI Evals › Software developers

Write a regression checklist before changing a review bot

Practice: Giving the right context. About 5 minutes.

About this exercise

You're about to ship v5 of your code-review bot's prompt, which caps comments at 3 per PR and adds an encouraging tone. Write a regression checklist of exactly 5 numbered lines, each 'N. INC-id | Re-run: the saved case | Pass if: ...'. It must include the incidents the 3-comment cap or the softer tone could bring back, and only incidents from the log. Plain text, under 170 words.

Common mistakes when you ask AI for this

  • Generic test ideas. The AI suggests 'test edge cases and tone' instead of re-running the saved cases you already have. Try: Say: "Each line names an incident id and its saved case file."
  • Missing the risky incidents. The AI doesn't connect the removed security line and the softer tone to old bugs coming back. Try: Add: "Include every incident the 3-comment cap or the softer tone could bring back."
  • Vague pass conditions. The AI writes 'bot behaves as expected', which tells a reviewer nothing. Try: Ask: "Pass if says the exact result, like 'flags the SQL injection'."
  • A full test plan. The AI writes sections on unit tests, load tests and monitoring. Try: Be exact: "Exactly 5 numbered lines in the shape INC-id | Re-run | Pass if."

What a strong prompt includes

  • Uses the incident log
  • Points at what the change risks
  • Asks for a pass condition
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises