Library › AI Evals › Product managers
Read a pull-request eval report and decide block or merge
- For Product managers
- You practice Catching mistakes and made-up facts
- Time about 5 minutes
- Level Advanced
About this exercise
You own the release gate for Pedalbot. Read the eval report for PR #412 and judge it against the merge bar in the report, not against the author's note. Write plain text: line 1 exactly 'Decision: BLOCK' or 'Decision: MERGE', then one line per bar rule formatted 'Rule name: value vs bar - PASS' or '- FAIL' (use the labels Overall accuracy, Safety failures, Biggest category drop), then 'Why: ' with one sentence under 25 words, then 'Next step: ' with a line ending exactly 'then rerun Safety and Refunds before merging.' Under 120 words.
Common mistakes when you ask AI for this
- Trusting the author's note. The AI sees overall above 90 and agrees the PR is ready. Try: Say: "Judge only against the three bar rules, not the author's note."
- Averaging the failures away. The AI treats one safety failure as small next to 183 passes. Try: Add: "Safety allows zero failures, so any failure is a FAIL."
- Skipping the exact lines. The AI writes a bold-headed essay with a table. Try: Give the line formats: "Decision: BLOCK or MERGE, then Rule: value vs bar - PASS or FAIL."
- Vague next step. The AI says 'investigate further'. Try: Add: "Next step names what to fix and which cases to rerun."
What a strong prompt includes
- Fixes line 1 as a decision
- Judges against the bar
- Asks for PASS or FAIL per rule
- Shows value vs bar
- Asks for plain text and a short length
Related exercises
- Accrue unpaid wages for the working days after the last pay period (Finance AI · Accountants)
- Age a returns queue in business days and flag the ones over the limit (Customer Support & Success AI · Customer support teams)
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Annualize recurring revenue without the one-time fee (Data AI · Finance and accounting)
- Answer a credit dispute with the full-hour rule and the math (Customer Support & Success AI · Customer support teams)