Library › AI Evals › Executives

Decide a release when one small number looks scary

  • For Executives
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Intermediate

About this exercise

You are the executive who signs off Pedalbot releases. Read the report for PR #431 and judge it only against the merge bar. Write plain text, exactly these lines: 'Decision: BLOCK' or 'Decision: MERGE'; 'Overall accuracy: value vs bar - PASS/FAIL'; 'Safety failures: value vs bar - PASS/FAIL'; 'Biggest drop (categories of 20+ cases): value vs bar - PASS/FAIL'; 'Watch item: Category before -> after (case ID)', written like 'Watch item: Pricing 9/9 -> 8/9 (P02)' for any small category that dropped; 'Why: ' one sentence under 20 words ending 'all three rules pass.' Under 110 words.

Common mistakes when you ask AI for this

  • Panicking at 12.5 points. The AI sees the biggest percentage drop and blocks, ignoring that the bar only counts categories of 20+ cases. Try: Say: "Apply the 20-case rule; smaller categories are watch items only."
  • Dropping the watch item. The AI says merge and never mentions the empty-message failure. Try: Add: "Add a Watch item line for any small category that dropped."
  • Essay instead of lines. The AI writes headings and a table of its own. Try: Add: "Plain text, exactly these six lines."
  • No numbers beside the bar. The AI says 'looks fine' without values. Try: Add: "Each bar line shows value vs bar."

What a strong prompt includes

  • Fixes the decision line
  • Applies the 20-case rule
  • Asks for value vs bar with PASS/FAIL
  • Asks for a watch item
  • Asks for plain text and a short length

Practice this now ↑

Related exercises