{"id":"eval-mgr-sampling-plan-ai-meeting-notes","audience":"managers","category":"evaluation","track":"evals","technique":"constraints","title":"Plan a monthly review of your team's AI meeting notes","free":false,"task":"You have 2 hours a month to review the AI notes your team's meetings produce, at 4 minutes per set of notes. Write exactly 4 numbered lines: Capacity (the math), Always review (the high-risk groups from the data, with their total), Random sample (how many of the rest, drawn across all teams), and Sample rate (the math as a percent). Plain text, under 100 words.","materialLabel":"Last month's meeting-notes numbers (attached to your prompt automatically)","material":"Customer Operations - AI meeting notes, October numbers\n\nTotal meetings with AI notes: 210\n\nHigh-risk groups (review all of these)\n- Meetings with external clients: 6\n- Notes an attendee flagged as wrong: 4\n\nEverything else: 200\n\nBy team (all 210)\nOnboarding 90 | Renewals 60 | Escalations 40 | Internal ops 20\n\nNotes\n- Last month I just read the notes from my own team's meetings (Onboarding). Never looked at Escalations.\n- Escalations meetings are where the AI mixed up two customers' names in September.\n- Review form is pass/fail per set of notes, takes about 4 minutes.\n- HR asked if we can keep notes for 90 days. Separate question.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Reviewing only your own team","why":"The AI accepts last month's habit of reading only Onboarding notes.","fix":"Say: \"Random sample drawn across all teams.\"","checks":["r2"]},{"title":"Forgetting the must-review notes","why":"The AI samples at random and misses client meetings and flagged notes.","fix":"Add: \"Always review the high-risk groups first, then sample the rest.\"","checks":["r1"]},{"title":"Hiding the math","why":"The AI says 'about 30' or '10%' without showing how.","fix":"Say: \"Show the math on each line, like 2 x 60 / 4 = 30.\"","checks":["r0","r2"]},{"title":"A long review policy","why":"The AI writes a review process with roles and tools.","fix":"Be exact: \"Exactly 4 numbered lines: Capacity, Always review, Random sample, Sample rate.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for the math","patterns":["math","show","x 60","calculat"]},{"label":"Reviews high-risk notes first","patterns":["high.?risk","always review","client","flagged"]},{"label":"Samples across teams","patterns":["random","all teams","across","every team"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A review plan has to fit your time and still see the risky notes. Start from capacity, cover the must-review groups, and sample the rest at random across every team.","rubricLabels":["Capacity shows 2 hours x 60 = 120 minutes / 4 = 30 sets of notes","Always review names client meetings (6) and attendee-flagged notes (4) for a total of 10, and Random sample gives 30 - 10 = 20 from the other 200","The random sample is drawn across all four teams, not only Onboarding, and Sample rate shows 20 / 200 = 10%"]}