{"id":"eval-pm-error-analysis-ai-meeting-notes-traces","audience":"product-managers","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 failing AI meeting-note traces into 3 failure types","free":false,"task":"You reviewed 13 failing outputs from your AI meeting-notes feature and noted what went wrong in each. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any trace where the recording itself was broken. Plain text, under 70 words.","materialLabel":"Your notes on the failing traces (attached to your prompt automatically)","material":"Huddlely - AI meeting notes, failing traces from Nov 3-5 (my notes)\n\n#1 action item owner says \"Sam\" but Priya took it\n#2 notes say the launch moved to Dec 1; nobody said that\n#3 decision \"go with vendor B\" missing from notes entirely\n#4 credited the pricing idea to Leo, it was Dana's\n#5 audio cut out after minute 4, nothing to summarize (recording bug)\n#6 notes say budget approved; meeting said \"approved pending finance\"... which isn't approved\n#7 the 3 action items at the end weren't captured\n#8 assigned the follow-up email to the host instead of Mo\n#9 invented a \"next meeting Tuesday\" line\n#10 the risk about the data migration wasn't mentioned\n#11 swapped who owns QA and who owns docs\n#12 said the client agreed to the price; client said \"we'll think about it\"\n#13 skipped the whole second agenda item\n\nContext: one fix next sprint. Eng thinks speaker detection is the main problem.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits wrong owners, wrong credit and swapped roles into separate groups.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes the problems without showing which traces belong where.","fix":"Show the shape: \"Type: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting the broken recording","why":"The AI counts #5, which failed because of the audio, not the AI.","fix":"Add: \"Leave out traces where the recording was broken, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out the broken recording","patterns":["#5","recording","audio","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis shows which failure is most common before you pick a fix. Fix the number of types, ask for ids behind every count, and drop traces the AI couldn't have gotten right.","rubricLabels":["Uses these 3 groups (names may differ): wrong person credited or assigned #1, #4, #8, #11 = 4; stated something that wasn't said or decided #2, #6, #9, #12 = 4; left out something that was said #3, #7, #10, #13 = 4","Leaves out #5 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}