{"id":"eval-pm-sampling-plan-ai-review-summaries","audience":"product-managers","category":"evaluation","track":"evals","technique":"constraints","title":"Plan a weekly human check of AI review summaries","free":false,"task":"Your product pages now show AI summaries of customer reviews, and your team can spend 4 hours a week checking them at 3 minutes each. Write exactly 4 numbered lines: Capacity (the math), Always review (the high-risk groups from the data, with their total), Random sample (how many of the rest, drawn across every category), and Sample rate (the math as a percent). Plain text, under 100 words.","materialLabel":"This week's summary numbers (attached to your prompt automatically)","material":"Larchmont Goods - AI review summaries on product pages, week of Nov 3\n\nSummaries updated this week: 2,530\n\nHigh-risk groups (policy: check every one)\n- Top 20 bestsellers (most traffic): 20\n- Summaries a shopper reported as misleading: 10\n\nEverything else: 2,500\n\nBy category (all 2,530)\nKitchen 900 | Outdoor 600 | Toys 480 | Beauty 350 | Electronics 200\n\nTeam notes\n- Last month we only checked Kitchen, since it's the biggest.\n- A Toys summary said \"safe for ages 2+\" when reviews said 5+. Caught by a parent.\n- Each check takes about 3 minutes (read reviews, compare summary).\n- Legal wants a quarterly report. Separate task.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Checking only the biggest category","why":"The AI accepts last month's habit of checking only Kitchen.","fix":"Say: \"Random sample drawn across every category.\"","checks":["r2"]},{"title":"Forgetting the must-check summaries","why":"The AI samples at random and misses bestsellers and shopper reports.","fix":"Add: \"Always review the high-risk groups first, then sample the rest.\"","checks":["r1"]},{"title":"Hiding the math","why":"The AI says 'about 80' or '2%' without showing how.","fix":"Say: \"Show the math on each line, like 4 x 60 / 3 = 80.\"","checks":["r0","r2"]},{"title":"A long QA process","why":"The AI writes a workflow with roles and dashboards.","fix":"Be exact: \"Exactly 4 numbered lines: Capacity, Always review, Random sample, Sample rate.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for the math","patterns":["math","show","x 60","calculat"]},{"label":"Reviews high-risk summaries first","patterns":["high.?risk","always review","bestseller","report"]},{"label":"Samples every category","patterns":["random","every category","all (five|5) categor","across"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A review plan has to fit your hours and still catch the risky summaries. Start from capacity, cover the must-check groups, and sample the rest at random across every category.","rubricLabels":["Capacity shows 4 hours x 60 = 240 minutes / 3 = 80 summaries","Always review names the top 20 bestsellers (20) and shopper-reported summaries (10) for a total of 30, and Random sample gives 80 - 30 = 50 from the other 2,500","The random sample is drawn across all five categories, not only Kitchen, and Sample rate shows 50 / 2,500 = 2%"]}