Library › AI Evals › Product managers
Check AI-written release notes against the shipped tickets
Practice: Catching mistakes and made-up facts. About 5 minutes.
About this exercise
An AI drafted release notes for version 4.2 from your ticket list, and you want to catch errors before customers see them. Write exactly 6 numbered lines, one per note, each labeled Shipped, Wrong or Not in this release. For each Wrong line, add the ticket's version in quotation marks. Judge only by the ticket list. Plain text, under 110 words.
Common mistakes when you ask AI for this
- Asking 'is this right?'. The AI says the notes look good and misses the delayed bank sync. Try: Say: "Label each note Shipped, Wrong or Not in this release."
- Missing the moved ticket. BB-315 is in the list, so the AI counts it as shipped even though it moved to v4.3. Try: Add: "A ticket moved to a later version is Not in this release."
- No proof for corrections. The AI says a note is wrong but doesn't show what the ticket says. Try: Add: "For Wrong, quote the ticket in quotation marks."
- A rewritten draft. The AI 'helps' by rewriting the release notes. Try: Be exact: "Exactly 6 numbered lines, one per note, nothing else."
What a strong prompt includes
- Gives exact labels
- Makes the tickets the judge
- Asks for a quote as evidence
- Says exactly how many
- Asks for plain text
Related exercises
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
- Approve part of an overtime request with the math (Writing AI · Managers and team leads)
- Catch a chatbot's mistake on your invoice total (Data AI · Freelancers and creators)
- Catch a chatbot's mistakes about a supplier's return policy (Research & AI Search · Small business owners)