{"id":"eval-pm-research-summary-vs-interview-notes","audience":"product-managers","category":"evaluation","track":"evals","technique":"verification","title":"Check an AI research summary against your interview notes","free":false,"task":"An AI summarized 4 customer interviews into 5 findings for your roadmap review, and you want to check each against your notes first. Write exactly 5 numbered lines, one per finding, each labeled Supported, Overstated or Not in notes. For each Overstated line, add what the notes actually say in quotation marks. Judge only by the notes. Plain text, under 110 words.","materialLabel":"Your interview notes and the AI's findings (attached to your prompt automatically)","material":"Pantrybox (meal-kit app) - interview notes, Oct 28-31\n\nI1 Rosa (2 kids): skips weeks often because \"the menu comes out too late, Thursday is too late to plan.\" Likes the 20-minute recipes.\nI2 Theo (single): cancelled once over price, came back for a discount. Says the app's skip button is \"hidden\".\nI3 Amina (couple): wants vegetarian options; says \"two veggie meals a week isn't enough.\" Loves the 20-minute recipes.\nI4 Grant (family of 5): portions too small for teens. Mentioned price \"a little\" but said he'd pay more for bigger portions.\n\nAI-WRITTEN FINDINGS\n1. Customers love the 20-minute recipes.\n2. All four customers say the price is too high.\n3. The skip button is hard to find.\n4. Customers want more vegetarian meals.\n5. Customers want a weekly grocery add-on feature.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Asking 'is it right?'","why":"The AI says the summary is 'a fair reflection' and misses the made-up finding.","fix":"Say: \"Label each finding Supported, Overstated or Not in notes.\"","checks":["r0","r2"]},{"title":"Letting 'all' slide","why":"One customer cancelled over price, and the AI accepts 'all four say price is too high'.","fix":"Add: \"Overstated means the finding claims more people or more strength than the notes show.\"","checks":["r0","r1"]},{"title":"No proof for corrections","why":"The AI says a finding is too strong without showing what the notes say.","fix":"Add: \"For Overstated, quote what the notes actually say.\"","checks":["r1"]},{"title":"A rewritten summary","why":"The AI 'helps' by rewriting the findings and adding recommendations.","fix":"Be exact: \"Exactly 5 numbered lines, nothing else.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives exact labels","patterns":["supported","overstated","not in notes","label"]},{"label":"Makes the notes the judge","patterns":["only.{0,30}(notes|interviews)","against (my|the) notes"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|finding)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI summaries of research turn 'one person said' into 'customers say'. Give the AI three labels, including one for overstated claims, and make your notes the only judge.","rubricLabels":["Labels finding 5 Not in notes, finding 2 Overstated, and findings 3 and 4 Overstated or Supported, with finding 1 Supported or Overstated (only 2 of 4 mentioned recipes)","Finding 2's line quotes or names what the notes say about price (only Theo cancelled over it, Grant mentioned it \"a little\" and would pay more), showing it was not all four","Uses only the three labels Supported, Overstated and Not in notes, with no rewritten findings, recommendations or intro"]}