Library › AI Evals › Product managers

Read a pull-request eval report and decide block or merge

  • For Product managers
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Advanced

About this exercise

You own the release gate for Pedalbot. Read the eval report for PR #412 and judge it against the merge bar in the report, not against the author's note. Write plain text: line 1 exactly 'Decision: BLOCK' or 'Decision: MERGE', then one line per bar rule formatted 'Rule name: value vs bar - PASS' or '- FAIL' (use the labels Overall accuracy, Safety failures, Biggest category drop), then 'Why: ' with one sentence under 25 words, then 'Next step: ' with a line ending exactly 'then rerun Safety and Refunds before merging.' Under 120 words.

Common mistakes when you ask AI for this

  • Trusting the author's note. The AI sees overall above 90 and agrees the PR is ready. Try: Say: "Judge only against the three bar rules, not the author's note."
  • Averaging the failures away. The AI treats one safety failure as small next to 183 passes. Try: Add: "Safety allows zero failures, so any failure is a FAIL."
  • Skipping the exact lines. The AI writes a bold-headed essay with a table. Try: Give the line formats: "Decision: BLOCK or MERGE, then Rule: value vs bar - PASS or FAIL."
  • Vague next step. The AI says 'investigate further'. Try: Add: "Next step names what to fix and which cases to rerun."

What a strong prompt includes

  • Fixes line 1 as a decision
  • Judges against the bar
  • Asks for PASS or FAIL per rule
  • Shows value vs bar
  • Asks for plain text and a short length

Practice this now ↑

Related exercises