Library › AI Evals › Software developers
Turn "the review bot is noisy" into 4 testable checks
Practice: Catching mistakes and made-up facts. About 5 minutes.
About this exercise
Engineers keep saying your AI code-review bot is 'noisy', and you need checks you can run against one review. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one review] - e.g. "quote from the thread"'. Don't use vague words like noisy, annoying or unhelpful as the test, and skip praise and feature requests. Plain text, under 130 words.
Common mistakes when you ask AI for this
- Keeping the vague words. The AI rewrites 'noisy' as 'comments should be relevant', which is just as hard to test. Try: Say: "Each check must be something you can see or count in one review."
- No example from real data. The AI writes generic review tips instead of tying checks to what engineers said. Try: Add: "Quote an example from the thread on each line."
- Treating every message as a problem. The AI turns Chen's praise and Fumi's feature idea into checks. Try: Add: "Skip praise and feature requests."
- Too many checks. The AI writes one check per message, plus some of its own. Try: Be exact: "Exactly 4 numbered checks."
What a strong prompt includes
- Asks for observable checks
- Asks for quoted examples
- Says exactly how many
- Skips praise and feature ideas
- Asks for plain text
Related exercises
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
- Approve part of an overtime request with the math (Writing AI · Managers and team leads)
- Brief a research agent to prep you for a sales call (Agents & Automation AI · Sales and customer success)
- Brief a research assistant to shortlist sources you can trust (Agents & Automation AI · College students)