{"id":"evals2-choose-cheapest-model-clears-bar","audience":"product-managers","category":"evaluation","track":"evals","stage":"choose-model","level":"intermediate","technique":"verification","title":"Pick the cheapest model that clears Pedalbot's 96% bar","free":false,"task":"Pedalbot's bar is 96% correct on the 200-question test set. Write 5 lines in this order: \"Bar: 96% of 200 = [number] correct\", then one line per model in the order Pebble-S, Cedar-M, Summit-L like \"Pebble-S: 184/200 = 92% - misses\" (use clears or misses), then \"Pick: [model] at [price] per 1,000 chats\". The pick is the cheapest model that clears the bar. Plain text, under 70 words.","materialLabel":"Test results (attached to your prompt automatically)","material":"Pedalbot model bake-off - 200-question test set (Nov 5)\n\nModel | Correct out of 200 | Price per 1,000 chats\nSummit-L | 197 | $55\nPebble-S | 184 | $6\nCedar-M | 193 | $14\n\nNotes\n- Same 200 questions and same prompt for all three.\n- Pebble-S answered all 60 hours questions correctly.\n- Summit-L had the best score of the three.\n- The bar of 96% was set from the cost of a wrong answer (see the bar worksheet).\n- Expected volume: 20,000 chats a month.\n- Rounding: show percentages to one decimal place only if needed.","rules":[{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Picking the best score","why":"The AI recommends Summit-L because it scored highest.","fix":"Say: \"Pick the cheapest model that clears the bar, not the best score.\"","checks":["r2"]},{"title":"Skipping the bar math","why":"The AI never works out how many correct answers 96% needs.","fix":"Add: \"Start with: Bar: 96% of 200 = N correct.\"","checks":["r0"]},{"title":"Wrong order or verdicts","why":"The AI sorts the models by score and drops the clears or misses.","fix":"Say: \"One line per model in the order Pebble-S, Cedar-M, Summit-L, ending clears or misses.\"","checks":["r1"]},{"title":"A comparison essay","why":"The AI writes tables, pros and cons and a long caveat.","fix":"Say: \"Exactly these 5 lines, plain text, under 70 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Asks for the bar math","patterns":["192|96% of 200|bar"]},{"label":"Wants cheapest that clears","patterns":["cheapest","clears?"]},{"label":"Fixes the model order","patterns":["pebble.{0,20}cedar.{0,20}summit"]},{"label":"Shows a line format","patterns":["/200 =","like"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"The right model is the cheapest one that still clears the bar, not the one with the top score. Ask for the bar in correct answers first, then check each model against it.","mission":"Choose the cheapest model that meets the bar.","musts":["Bar worked out first","One verdict line per model","Pick cheapest that clears"],"rubricLabels":["The first line shows the bar as 96% of 200 = 192 correct","The three model lines are in the order Pebble-S, Cedar-M, Summit-L and show 184/200 = 92% misses, 193/200 = 96.5% clears and 197/200 = 98.5% clears","The last line is \"Pick: Cedar-M at $14 per 1,000 chats\" and no other model is picked"]}