{"id":"eval-dev-regression-checklist-review-bot-v5","audience":"developers","category":"evaluation","track":"evals","technique":"context","title":"Write a regression checklist before changing a review bot","free":false,"task":"You're about to ship v5 of your code-review bot's prompt, which caps comments at 3 per PR and adds an encouraging tone. Write a regression checklist of exactly 5 numbered lines, each 'N. INC-id | Re-run: the saved case | Pass if: ...'. It must include the incidents the 3-comment cap or the softer tone could bring back, and only incidents from the log. Plain text, under 170 words.","materialLabel":"The prompt change and the incident log (attached to your prompt automatically)","material":"reviewbot - prompt change v4 -> v5 (PR #882)\n\nDIFF (system prompt)\n+ Leave at most 3 comments per PR. Pick the most important.\n+ Use an encouraging, positive tone.\n- (removed) Always flag security issues, even on large PRs.\n\nINCIDENT LOG (each has a saved replay case in evals/cases/)\nINC-12: On a 1,400-line PR the bot missed a SQL injection in users.py. Fixed in v3 by adding \"Always flag security issues, even on large PRs.\" Case: inc12_big_pr.json\nINC-19: Bot pasted an API key from .env into a comment. Fixed in v2 (redaction). Case: inc19_secret.json\nINC-23: Bot reviewed in Spanish on an English repo. Fixed in v4. Case: inc23_lang.json\nINC-27: Bot said \"Looks great, ship it!\" on a PR whose tests were failing. Fixed in v4 by requiring it to read CI status. Case: inc27_red_ci.json\nINC-30: Bot left 41 whitespace nitpicks on one PR. v5's cap is meant to fix this. Case: inc30_nits.json\nINC-33: Bot crashed on binary files (.png). Fixed in code, not the prompt. Case: inc33_binary.json\n\nTeam chat\n- Sam: \"can we also add a test for Rust support?\" (we don't support Rust yet)\n- Ana: \"the removed security line was redundant I think\"\n- Release is Friday.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":170}],"mistakes":[{"title":"Generic test ideas","why":"The AI suggests 'test edge cases and tone' instead of re-running the saved cases you already have.","fix":"Say: \"Each line names an incident id and its saved case file.\"","checks":["r1"]},{"title":"Missing the risky incidents","why":"The AI doesn't connect the removed security line and the softer tone to old bugs coming back.","fix":"Add: \"Include every incident the 3-comment cap or the softer tone could bring back.\"","checks":["r0"]},{"title":"Vague pass conditions","why":"The AI writes 'bot behaves as expected', which tells a reviewer nothing.","fix":"Ask: \"Pass if says the exact result, like 'flags the SQL injection'.\"","checks":["r2"]},{"title":"A full test plan","why":"The AI writes sections on unit tests, load tests and monitoring.","fix":"Be exact: \"Exactly 5 numbered lines in the shape INC-id | Re-run | Pass if.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Uses the incident log","patterns":["incident","INC","saved case","replay"]},{"label":"Points at what the change risks","patterns":["cap","tone","security","could (break|bring)"]},{"label":"Asks for a pass condition","patterns":["pass if","expected","specific"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|item|check)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Before a prompt change, re-run the old failures the change could bring back. Point the AI at the incident log and the diff, and ask for a concrete pass condition per case.","rubricLabels":["Includes INC-12 (the security line was removed and the cap could drop the security comment), INC-27 (the encouraging tone could approve failing tests) and INC-30 (the cap should now pass)","Every line names a real incident id and its saved case file from the log (for example inc12_big_pr.json), and there is no line about Rust or anything not in the log","Every Pass if is a specific observable result (for example 'flags the SQL injection in users.py' or 'does not approve while CI is red'), not 'works correctly'"]}