Library › AI Evals › Software developers

Turn "the review bot is noisy" into 4 testable checks

Practice: Catching mistakes and made-up facts. About 5 minutes.

About this exercise

Engineers keep saying your AI code-review bot is 'noisy', and you need checks you can run against one review. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one review] - e.g. "quote from the thread"'. Don't use vague words like noisy, annoying or unhelpful as the test, and skip praise and feature requests. Plain text, under 130 words.

Common mistakes when you ask AI for this

  • Keeping the vague words. The AI rewrites 'noisy' as 'comments should be relevant', which is just as hard to test. Try: Say: "Each check must be something you can see or count in one review."
  • No example from real data. The AI writes generic review tips instead of tying checks to what engineers said. Try: Add: "Quote an example from the thread on each line."
  • Treating every message as a problem. The AI turns Chen's praise and Fumi's feature idea into checks. Try: Add: "Skip praise and feature requests."
  • Too many checks. The AI writes one check per message, plus some of its own. Try: Be exact: "Exactly 4 numbered checks."

What a strong prompt includes

  • Asks for observable checks
  • Asks for quoted examples
  • Says exactly how many
  • Skips praise and feature ideas
  • Asks for plain text

Practice this now ↑

Related exercises