Library › AI Evals › Managers and team leads

Decide ship or rerun when a score rose four points

  • For Managers and team leads
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Advanced

About this exercise

Pedalbot's eval score went from 78% to 82% after a prompt change, and the CEO wants to ship. The material also has repeat runs of the old prompt. Write a plain text reply: the first line is 'Decision: Ship' or 'Decision: Rerun' followed by the math comparing the rise in points to the old prompt's spread (highest minus lowest run). Then exactly 3 checklist lines, each starting with '- [ ]', which include how many reruns to do and one numeric ship condition. Under 95 words.

Common mistakes when you ask AI for this

  • Shipping on the headline. The AI agrees 82 beats 80 and ships. Try: Say: "Compare the rise to how much the old prompt varied on its own."
  • Comparing to one old run. The AI compares 82 only to Tuesday's 78. Try: Add: "Use the spread of the repeat runs: highest minus lowest."
  • A vague next step. The AI says 'run more tests' with no number. Try: Say: "Give the number of reruns and a numeric ship condition."
  • Wrong checklist shape. The AI writes paragraphs or a different bullet style. Try: Say: "Exactly 3 lines starting with - [ ]."
  • Markdown and length. AI defaults to bold headings and long reasoning. Try: Ask for "plain text, under 95 words."

What a strong prompt includes

  • Compares the rise to the spread
  • Uses highest minus lowest
  • Asks for a decision first
  • Asks for rerun count and ship condition
  • Asks for plain text

Practice this now ↑

Related exercises