{"id":"evals4-merge-with-watch-item-greeting-pr","audience":"executives","category":"evaluation","track":"evals","stage":"keep-honest","technique":"verification","level":"intermediate","title":"Decide a release when one small number looks scary","free":false,"task":"You are the executive who signs off Pedalbot releases. Read the report for PR #431 and judge it only against the merge bar. Write plain text, exactly these lines: 'Decision: BLOCK' or 'Decision: MERGE'; 'Overall accuracy: value vs bar - PASS/FAIL'; 'Safety failures: value vs bar - PASS/FAIL'; 'Biggest drop (categories of 20+ cases): value vs bar - PASS/FAIL'; 'Watch item: Category before -> after (case ID)', written like 'Watch item: Pricing 9/9 -> 8/9 (P02)' for any small category that dropped; 'Why: ' one sentence under 20 words ending 'all three rules pass.' Under 110 words.","materialLabel":"PR eval report (attached to your prompt automatically)","material":"Eval report - Pull request #431 \"Warmer Pedalbot greeting\"\nRun: 200 cases, same suite for both versions, same judge version.\n\nCategory      Cases   Main passes   PR passes   Main %   PR %\nRefunds         50        45           45        90.0    90.0\nAvailability    50        47           47        94.0    94.0\nPricing         40        38           38        95.0    95.0\nSafety          30        30           30       100.0   100.0\nGreeting tone   22        20           21        90.9    95.5\nOdd input        8         8            7       100.0    87.5\nTOTAL          200       188          188        94.0    94.0\n\nThe one new failure: case E01 (empty message). The new greeting answered \"Hi there!\" and did not ask what the rider needs.\n\nMerge bar (from the team handbook, v3):\n1. Overall accuracy at least 90.0 percent.\n2. Zero failures in the Safety category.\n3. No category with 20 or more cases may drop more than 3.0 points against main. Smaller categories are listed as watch items only.","rules":[{"type":"plainText"},{"type":"maxWords","max":110}],"promptHints":[{"label":"Fixes the decision line","patterns":["decision: (block|merge)","first line","line 1"]},{"label":"Applies the 20-case rule","patterns":["20 or more","20\\+","20 cases","smaller categor","watch item"]},{"label":"Asks for value vs bar with PASS/FAIL","patterns":["vs","pass","fail"]},{"label":"Asks for a watch item","patterns":["watch"]},{"label":"Asks for plain text and a short length","patterns":["plain","no bold","under \\d+ words"]}],"mistakes":[{"title":"Panicking at 12.5 points","why":"The AI sees the biggest percentage drop and blocks, ignoring that the bar only counts categories of 20+ cases.","fix":"Say: \"Apply the 20-case rule; smaller categories are watch items only.\"","checks":["r0","r1"]},{"title":"Dropping the watch item","why":"The AI says merge and never mentions the empty-message failure.","fix":"Add: \"Add a Watch item line for any small category that dropped.\"","checks":["r2"]},{"title":"Essay instead of lines","why":"The AI writes headings and a table of its own.","fix":"Add: \"Plain text, exactly these six lines.\"","checks":["plainText","maxWords"]},{"title":"No numbers beside the bar","why":"The AI says 'looks fine' without values.","fix":"Add: \"Each bar line shows value vs bar.\"","checks":["r1"]}],"hint":"Not every red number is a blocker. Tell the AI to apply the bar exactly as written, including the size rule, and to keep small drops as watch items.","mission":"Decide merge or block, keeping small drops as watch items.","musts":["Apply the 20-case rule","Show value vs bar","Include a watch item"],"rubricLabels":["Line 1 is exactly 'Decision: MERGE' and the reply does not recommend blocking or delaying the merge","The three bar lines are right: Overall accuracy 94.0% vs 90.0% - PASS; Safety failures 0 vs 0 - PASS; Biggest drop in categories of 20+ cases is 0.0 points (nothing dropped) vs 3.0 - PASS","'Watch item:' is exactly 'Watch item: Odd input 8/8 -> 7/8 (E01)' and the Why line ends 'all three rules pass.'"]}