{"id":"evals4-block-or-merge-shorter-answers-pr","audience":"product-managers","category":"evaluation","track":"evals","stage":"keep-honest","technique":"verification","level":"advanced","title":"Read a pull-request eval report and decide block or merge","free":false,"task":"You own the release gate for Pedalbot. Read the eval report for PR #412 and judge it against the merge bar in the report, not against the author's note. Write plain text: line 1 exactly 'Decision: BLOCK' or 'Decision: MERGE', then one line per bar rule formatted 'Rule name: value vs bar - PASS' or '- FAIL' (use the labels Overall accuracy, Safety failures, Biggest category drop), then 'Why: ' with one sentence under 25 words, then 'Next step: ' with a line ending exactly 'then rerun Safety and Refunds before merging.' Under 120 words.","materialLabel":"PR eval report (attached to your prompt automatically)","material":"Eval report - Pull request #412 \"Shorter Pedalbot answers\"\nRun: 200 cases, same suite for both versions. Author's note: \"Overall is still above 90, ready to merge.\"\n\nCategory      Cases   Main passes   PR passes   Main %   PR %\nRefunds         50        45           42        90.0    84.0\nAvailability    50        47           47        94.0    94.0\nPricing         40        38           37        95.0    92.5\nSafety          30        30           29       100.0    96.7\nHandoff         30        27           28        90.0    93.3\nTOTAL          200       187          183        93.5    91.5\n\nFailed safety case S-17: rider said \"I fell and my wrist hurts.\" The shorter PR reply gave a bike-fit tip and did not mention emergency help.\nRefunds: 8 failures now versus 5 on main. 3 of the 8 say \"full refund\" where policy gives 50 percent inside 24 hours.\nJudge: same judge version for both runs.\n\nMerge bar (from the team handbook, v3):\n1. Overall accuracy at least 90.0 percent.\n2. Zero failures in the Safety category.\n3. No category may drop more than 3.0 points against main.","rules":[{"type":"plainText"},{"type":"maxWords","max":120}],"promptHints":[{"label":"Fixes line 1 as a decision","patterns":["decision: (block|merge)","first line","line 1"]},{"label":"Judges against the bar","patterns":["bar","merge rule","handbook","rules? in the report"]},{"label":"Asks for PASS or FAIL per rule","patterns":["pass","fail"]},{"label":"Shows value vs bar","patterns":["vs","versus","value"]},{"label":"Asks for plain text and a short length","patterns":["plain","no bold","under \\d+ words"]}],"mistakes":[{"title":"Trusting the author's note","why":"The AI sees overall above 90 and agrees the PR is ready.","fix":"Say: \"Judge only against the three bar rules, not the author's note.\"","checks":["r0","r1"]},{"title":"Averaging the failures away","why":"The AI treats one safety failure as small next to 183 passes.","fix":"Add: \"Safety allows zero failures, so any failure is a FAIL.\"","checks":["r1"]},{"title":"Skipping the exact lines","why":"The AI writes a bold-headed essay with a table.","fix":"Give the line formats: \"Decision: BLOCK or MERGE, then Rule: value vs bar - PASS or FAIL.\"","checks":["plainText","maxWords","r0"]},{"title":"Vague next step","why":"The AI says 'investigate further'.","fix":"Add: \"Next step names what to fix and which cases to rerun.\"","checks":["r2"]}],"hint":"The decision comes from the written bar, line by line. Ask for one PASS or FAIL per rule with the number next to the bar it is measured against.","mission":"Decide block or merge for the pull request.","musts":["Judge against stated bar","One PASS/FAIL per rule","Name the failing case"],"rubricLabels":["Line 1 is exactly 'Decision: BLOCK' and there is no 'MERGE' recommendation or merge-now-fix-later suggestion anywhere","The three bar lines use the right numbers: Overall accuracy 91.5% vs 90.0% bar - PASS; Safety failures 1 vs 0 - FAIL; Biggest category drop Refunds 6.0 points vs 3.0 - FAIL","'Why:' names the failed safety case S-17 (injured rider, no emergency help) and the Refunds drop, and the 'Next step:' line ends exactly 'then rerun Safety and Refunds before merging.'"]}