Library › AI Evals › Product managers
Pick the cheapest model that clears Pedalbot's 96% bar
- For Product managers
- You practice Catching mistakes and made-up facts
- Time about 5 minutes
- Level Intermediate
About this exercise
Pedalbot's bar is 96% correct on the 200-question test set. Write 5 lines in this order: "Bar: 96% of 200 = [number] correct", then one line per model in the order Pebble-S, Cedar-M, Summit-L like "Pebble-S: 184/200 = 92% - misses" (use clears or misses), then "Pick: [model] at [price] per 1,000 chats". The pick is the cheapest model that clears the bar. Plain text, under 70 words.
Common mistakes when you ask AI for this
- Picking the best score. The AI recommends Summit-L because it scored highest. Try: Say: "Pick the cheapest model that clears the bar, not the best score."
- Skipping the bar math. The AI never works out how many correct answers 96% needs. Try: Add: "Start with: Bar: 96% of 200 = N correct."
- Wrong order or verdicts. The AI sorts the models by score and drops the clears or misses. Try: Say: "One line per model in the order Pebble-S, Cedar-M, Summit-L, ending clears or misses."
- A comparison essay. The AI writes tables, pros and cons and a long caveat. Try: Say: "Exactly these 5 lines, plain text, under 70 words."
What a strong prompt includes
- Asks for the bar math
- Wants cheapest that clears
- Fixes the model order
- Shows a line format
- Asks for plain text
Related exercises
- Accrue unpaid wages for the working days after the last pay period (Finance AI · Accountants)
- Age a returns queue in business days and flag the ones over the limit (Customer Support & Success AI · Customer support teams)
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Annualize recurring revenue without the one-time fee (Data AI · Finance and accounting)
- Answer a credit dispute with the full-hour rule and the math (Customer Support & Success AI · Customer support teams)