{"id":"evals2-choose-prompt-first-plan","audience":"developers","category":"evaluation","track":"evals","stage":"choose-model","level":"intermediate","technique":"verification","title":"Write a prompt-first plan when the team says \"just upgrade\"","free":false,"task":"A teammate says \"just upgrade Pedalbot to Summit-L.\" Using the failure breakdown, write a plan as a checklist of exactly 4 lines in this order, each starting \"- [ ]\": fix the prompt, load the missing pricing sheet, correct the bad test cases, then rerun all 200 cases. Each of the first three lines ends with how many failures it fixes, like \"(fixes 22 of 40)\". The rerun line gives the failures left and the new score, like \"expect 6 left, 194/200 = 97%\". Then a last line \"Upgrade only if: ...\" using the 96% bar. Plain text, under 120 words.","materialLabel":"Slack thread and failure breakdown (attached to your prompt automatically)","material":"#pedalbot-eval thread, Nov 6\n\nJo (lead): Pass rate is only 80%. Just upgrade to Summit-L. It's 4x the price but it'll fix everything.\n\nRaj: I went through all 40 failures on the 200-question set. Breakdown:\n- 22 failures: the reply ignores the house rules (keep it under 60 words, always include the booking link). The prompt never states either rule.\n- 10 failures: wrong prices. The bot is reading pricing sheet v3; v4 was never loaded into its context.\n- 6 failures: multi-step date math (for example, a pickup 3 days before a holiday weekend).\n- 2 failures: test cases where the expected answer is wrong (typos in the golden answers).\n\nThe bar is 96% on this set, from the cost-of-a-wrong-answer worksheet.\n\nJo: Summit-L got 190/200 on the same set when I tried it.\nRaj: With the same weak prompt and the old pricing sheet, yes.\n\nThe next release is Nov 17.","rules":[{"type":"maxBullets","max":4},{"type":"plainText"},{"type":"maxWords","max":120}],"mistakes":[{"title":"Agreeing to upgrade","why":"The AI takes the lead's word and recommends the bigger model.","fix":"Say: \"Fix the causes you can see in the failures before changing the model.\"","checks":["r2"]},{"title":"Missing counts","why":"The AI lists steps without saying how many failures each fixes.","fix":"Add: \"End each of the first three lines with (fixes N of 40).\"","checks":["r0"]},{"title":"No new score","why":"The AI never works out what the rerun should score.","fix":"Add: \"Rerun line: expect 6 left, 194/200 = 97%.\"","checks":["r1"]},{"title":"No upgrade trigger","why":"The AI ends with a vague 'consider upgrading later'.","fix":"Add: \"Last line: Upgrade only if the rerun has more than 8 failures.\"","checks":["r2"]},{"title":"A long plan","why":"The AI writes sections, bold headings and extra bullets.","fix":"Say: \"Exactly 4 checklist lines plus one last line, plain text.\"","checks":["maxBullets","plainText","maxWords"]}],"promptHints":[{"label":"Asks for a checklist","patterns":["checklist","- \\[ \\]"]},{"label":"Fixes the prompt first","patterns":["prompt","before (the )?upgrade|first"]},{"label":"Counts failures per step","patterns":["fixes? \\d+|of 40|failures?"]},{"label":"Sets the upgrade trigger","patterns":["upgrade only if","more than 8|bar"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"Most failures here have simple causes in the prompt, the context and the tests. Ask for a checklist that fixes those first and says exactly when an upgrade would be justified.","mission":"Plan the fixes before agreeing to a model upgrade.","musts":["Four checklist lines","Failures fixed per step","Upgrade-only-if line"],"rubricLabels":["The checklist has exactly 4 lines starting \"- [ ]\", in this order: fix the prompt (fixes 22 of 40), load pricing sheet v4 (fixes 10), correct the 2 bad test cases (fixes 2), rerun all 200 cases","The rerun line says expect 6 failures left and 194/200 = 97%, which clears the 96% bar","The last line starts \"Upgrade only if:\" and says the rerun still has more than 8 failures (the 96% bar on 200), and the plan does not recommend upgrading now"]}