{"id":"evals3-ship-or-rerun-four-point-rise","audience":"managers","category":"evaluation","track":"evals","stage":"fix-prompt","level":"advanced","technique":"verification","title":"Decide ship or rerun when a score rose four points","free":false,"task":"Pedalbot's eval score went from 78% to 82% after a prompt change, and the CEO wants to ship. The material also has repeat runs of the old prompt. Write a plain text reply: the first line is 'Decision: Ship' or 'Decision: Rerun' followed by the math comparing the rise in points to the old prompt's spread (highest minus lowest run). Then exactly 3 checklist lines, each starting with '- [ ]', which include how many reruns to do and one numeric ship condition. Under 95 words.","materialLabel":"Eval run history and notes (attached to your prompt automatically)","material":"SPOKEWISE - PEDALBOT EVAL RUNS (same 100 test cases each time, pass rate)\n\nOld prompt (v6, unchanged) run on different days:\nMon 74%\nTue 78%\nWed 81%\nThu 77%\n\nNew prompt (v7), one run, Fri: 82%\n\nTarget set for launch: 80%.\n\nNotes\n- The model samples its answers, so the same prompt on the same 100 cases doesn't always land on the same score.\n- Maya (CEO): \"82 beats our 80 target and it's higher than Tuesday's 78. Ship v7 today.\"\n- A full run of the 100 cases takes about 6 minutes and costs a few dollars.\n- Shipping v7 means every Spokewise customer sees it on Monday.\n- Engineering says v7 is \"a bit shorter and clearer\" than v6; no logic changed.","rules":[{"type":"maxBullets","max":3},{"type":"plainText"},{"type":"maxWords","max":95}],"mistakes":[{"title":"Shipping on the headline","why":"The AI agrees 82 beats 80 and ships.","fix":"Say: \"Compare the rise to how much the old prompt varied on its own.\"","checks":["r0","r1"]},{"title":"Comparing to one old run","why":"The AI compares 82 only to Tuesday's 78.","fix":"Add: \"Use the spread of the repeat runs: highest minus lowest.\"","checks":["r0"]},{"title":"A vague next step","why":"The AI says 'run more tests' with no number.","fix":"Say: \"Give the number of reruns and a numeric ship condition.\"","checks":["r2"]},{"title":"Wrong checklist shape","why":"The AI writes paragraphs or a different bullet style.","fix":"Say: \"Exactly 3 lines starting with - [ ].\"","checks":["maxBullets","r2"]},{"title":"Markdown and length","why":"AI defaults to bold headings and long reasoning.","fix":"Ask for \"plain text, under 95 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Compares the rise to the spread","patterns":["spread","range","variation","variance","noise"]},{"label":"Uses highest minus lowest","patterns":["highest","lowest","81 ?- ?74","7 points"]},{"label":"Asks for a decision first","patterns":["first line","decision:","rerun or ship"]},{"label":"Asks for rerun count and ship condition","patterns":["rerun","at least \\d","ship (condition|only)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Two runs of the same prompt can differ by several points. Measure how much the old prompt varied by itself before you call a smaller rise real.","mission":"Decide whether a 4-point rise is real before shipping.","musts":["Decision line first","Compare rise to spread","Exactly 3 checklist items"],"rubricLabels":["The first line is 'Decision: Rerun' and shows the math: the rise is 82 - 78 = 4 points, while the old prompt's runs spread 81 - 74 = 7 points, so the rise is within normal run-to-run variation","Does not treat 82% beating the 80% target as proof, noting the old prompt itself reached 81% on one day","Has exactly 3 checklist lines starting '- [ ]': one says to rerun v7 at least 3 more times on the same 100 cases, and one gives a numeric ship condition (such as every v7 run above 81%)"]}