{"id":"eval-cs-human-review-sampling-plan-math","audience":"customer-support","category":"evaluation","track":"evals","technique":"constraints","title":"Plan which bot chats your team reviews each week","free":false,"task":"Your team can spend 5 hours a week reviewing support bot chats at 3 minutes per chat. Write exactly 4 numbered lines: Capacity (the math), Always review (the high-risk groups from the data, with their total), Random sample (how many of the remaining chats, drawn at random across the whole week), and Sample rate (the math as a percent). Plain text, under 100 words.","materialLabel":"This week's chat volume and team notes (attached to your prompt automatically)","material":"Cobalt Bay Mobile - support bot weekly numbers (week of Nov 2)\n\nTotal bot chats: 2,550\n\nHigh-risk groups (from the incident policy: review every one)\n- Chats where the bot issued a refund over $200: 30\n- Chats the customer reopened within 24 hours: 20\n\nEverything else: 2,500 chats\n\nVolume by day (all chats)\nMon 610 | Tue 420 | Wed 380 | Thu 390 | Fri 450 | Sat 160 | Sun 140\n\nTeam notes\n- Reviewers: Ade and Wen, 2.5 hours each per week.\n- Average review time: 3 minutes per chat using the QA form.\n- Wen: \"Last month we just read the first 100 chats on Monday morning. Easy to pull.\"\n- Problem from that: weekend chats (when the bot handles SIM swaps alone) were never reviewed.\n- Ade wants to add a 1-10 'chat quality' score to the form. Lead said keep it pass/fail.\n- Next month the bot starts handling plan upgrades too.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Sampling the easy way","why":"The AI accepts 'read the first 100 on Monday', which never covers weekend chats.","fix":"Say: \"Random sample drawn across the whole week, weekend included.\"","checks":["r2"]},{"title":"Forgetting the must-review chats","why":"The AI samples everything randomly and misses the high-risk groups your policy covers.","fix":"Add: \"Always review the high-risk groups first, then sample the rest.\"","checks":["r1"]},{"title":"Hiding the math","why":"The AI states '100 chats' or '2%' without showing how it got there.","fix":"Say: \"Show the math on each line, like 5 x 60 / 3 = 100.\"","checks":["r0","r2"]},{"title":"A QA strategy essay","why":"By default the AI writes a long plan with scoring rubrics and tools.","fix":"Be exact: \"Exactly 4 numbered lines: Capacity, Always review, Random sample, Sample rate.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for the math","patterns":["math","show","x 60","calculat"]},{"label":"Reviews high-risk chats first","patterns":["high.?risk","always review","refund","reopen"]},{"label":"Samples the whole week","patterns":["random","whole week","weekend","all (7|seven) days"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A review plan has to fit your time and still catch the risky chats. Ask the AI to start from capacity, cover the must-review groups, and sample the rest at random across the week.","rubricLabels":["Capacity shows 5 hours x 60 = 300 minutes / 3 = 100 chats","Always review names refunds over $200 (30) and chats reopened within 24 hours (20) for a total of 50, and Random sample gives 100 - 50 = 50 chats from the other 2,500","The random sample is drawn across all 7 days including the weekend (not the first chats on Monday), and Sample rate shows 50 / 2,500 = 2%"]}