{"id":"eval-dev-noisy-review-bot-feedback-to-checks","audience":"developers","category":"evaluation","track":"evals","technique":"verification","title":"Turn \"the review bot is noisy\" into 4 testable checks","free":false,"task":"Engineers keep saying your AI code-review bot is 'noisy', and you need checks you can run against one review. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one review] - e.g. \"quote from the thread\"'. Don't use vague words like noisy, annoying or unhelpful as the test, and skip praise and feature requests. Plain text, under 130 words.","materialLabel":"The engineering Slack thread (attached to your prompt automatically)","material":"#eng-tools thread, Wed\n\nAri: the review bot is so noisy\nBea: it left 23 comments on my 40-line PR\nChen: half of them are \"consider adding a comment here\"\nAri: it flagged formatting that our linter already auto-fixes\nDev: it comments on files I didn't touch, like it's reviewing the whole repo\nBea: and it repeats the same comment on every function. same text 6 times\nChen: honestly it caught a real null bug last week, that was great\nFumi: can it also write the PR description for me?\nDev: also it suggests renaming variables to names that already exist in the file\nAri: noisy noisy noisy\n\nContext: 60 engineers, the bot reviews every PR. You're tuning its prompt next sprint and want a way to grade 50 reviews before and after.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":130}],"mistakes":[{"title":"Keeping the vague words","why":"The AI rewrites 'noisy' as 'comments should be relevant', which is just as hard to test.","fix":"Say: \"Each check must be something you can see or count in one review.\"","checks":["r0"]},{"title":"No example from real data","why":"The AI writes generic review tips instead of tying checks to what engineers said.","fix":"Add: \"Quote an example from the thread on each line.\"","checks":["r1"]},{"title":"Treating every message as a problem","why":"The AI turns Chen's praise and Fumi's feature idea into checks.","fix":"Add: \"Skip praise and feature requests.\"","checks":["r2"]},{"title":"Too many checks","why":"The AI writes one check per message, plus some of its own.","fix":"Be exact: \"Exactly 4 numbered checks.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for observable checks","patterns":["see or count","observable","countable","testable"]},{"label":"Asks for quoted examples","patterns":["quote","example","e\\.g\\."]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(check|line|criteri)","exactly \\d+"]},{"label":"Skips praise and feature ideas","patterns":["praise","feature","pr description","real bug"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"'Noisy' can't be graded, but 'comments on files the PR didn't touch' can. Ask the AI to turn each complaint into something you can see or count in one review, with a real quote.","rubricLabels":["Each check's test is something visible or countable in one review (a comment count per lines changed, comments on unchanged files, the same comment text repeated, style issues the linter already fixes, generic 'consider adding a comment' notes), not a judgment word like noisy, annoying or unhelpful","Each of the 4 lines includes an example quoted from the thread, such as \"23 comments on my 40-line PR\" or \"same text 6 times\"","No check penalizes catching real bugs, and no check is about writing PR descriptions"]}