{"id":"eval-pm-release-notes-vs-tickets","audience":"product-managers","category":"evaluation","track":"evals","technique":"verification","title":"Check AI-written release notes against the shipped tickets","free":false,"task":"An AI drafted release notes for version 4.2 from your ticket list, and you want to catch errors before customers see them. Write exactly 6 numbered lines, one per note, each labeled Shipped, Wrong or Not in this release. For each Wrong line, add the ticket's version in quotation marks. Judge only by the ticket list. Plain text, under 110 words.","materialLabel":"The ticket list and the AI's draft (attached to your prompt automatically)","material":"Brookline Budget app - v4.2 release, ticket export (Nov 5)\n\nBB-301 Dark mode for iOS. Status: Done.\nBB-308 Export transactions to CSV. Status: Done. Note: CSV only, PDF export is BB-340.\nBB-312 Shared budgets for up to 4 people. Status: Done.\nBB-315 Bank sync for credit unions. Status: Moved to v4.3 (vendor delay).\nBB-322 Faster app launch, about 30% quicker on Android. Status: Done.\nBB-340 PDF export. Status: Not started.\n\nAI-DRAFTED RELEASE NOTES (v4.2)\n1. Dark mode is here for iPhone users.\n2. Export your transactions to CSV or PDF.\n3. Share a budget with up to 6 people.\n4. Credit union accounts now sync automatically.\n5. The Android app opens about 30% faster.\n6. New spending insights show where your money goes each month.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Asking 'is this right?'","why":"The AI says the notes look good and misses the delayed bank sync.","fix":"Say: \"Label each note Shipped, Wrong or Not in this release.\"","checks":["r0","r2"]},{"title":"Missing the moved ticket","why":"BB-315 is in the list, so the AI counts it as shipped even though it moved to v4.3.","fix":"Add: \"A ticket moved to a later version is Not in this release.\"","checks":["r0"]},{"title":"No proof for corrections","why":"The AI says a note is wrong but doesn't show what the ticket says.","fix":"Add: \"For Wrong, quote the ticket in quotation marks.\"","checks":["r1"]},{"title":"A rewritten draft","why":"The AI 'helps' by rewriting the release notes.","fix":"Be exact: \"Exactly 6 numbered lines, one per note, nothing else.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives exact labels","patterns":["shipped","wrong","not in this release","label"]},{"label":"Makes the tickets the judge","patterns":["only.{0,30}(ticket|list|export)","against the (ticket|list)"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(6|six)\\b.{0,25}(line|note)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI release notes often include work that slipped or round a number up. Give the AI three labels and make the ticket list the only judge.","rubricLabels":["Labels notes 1 and 5 Shipped, notes 2 and 3 Wrong, and notes 4 and 6 Not in this release","The Wrong lines give the ticket's version in quotation marks: note 2 says CSV only (PDF is not done), and note 3 says up to 4 people","Uses only the three labels Shipped, Wrong and Not in this release, with no rewritten notes, marketing copy or intro"]}