Library › AI Evals › Product managers

Pick the cheapest model that clears Pedalbot's 96% bar

  • For Product managers
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Pedalbot's bar is 96% correct on the 200-question test set. Write 5 lines in this order: "Bar: 96% of 200 = [number] correct", then one line per model in the order Pebble-S, Cedar-M, Summit-L like "Pebble-S: 184/200 = 92% - misses" (use clears or misses), then "Pick: [model] at [price] per 1,000 chats". The pick is the cheapest model that clears the bar. Plain text, under 70 words.

Common mistakes when you ask AI for this

  • Picking the best score. The AI recommends Summit-L because it scored highest. Try: Say: "Pick the cheapest model that clears the bar, not the best score."
  • Skipping the bar math. The AI never works out how many correct answers 96% needs. Try: Add: "Start with: Bar: 96% of 200 = N correct."
  • Wrong order or verdicts. The AI sorts the models by score and drops the clears or misses. Try: Say: "One line per model in the order Pebble-S, Cedar-M, Summit-L, ending clears or misses."
  • A comparison essay. The AI writes tables, pros and cons and a long caveat. Try: Say: "Exactly these 5 lines, plain text, under 70 words."

What a strong prompt includes

  • Asks for the bar math
  • Wants cheapest that clears
  • Fixes the model order
  • Shows a line format
  • Asks for plain text

Practice this now ↑

Related exercises