{"id":"eval-mgr-error-analysis-ai-weekly-reports","audience":"managers","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 flawed AI weekly reports into 3 failure types","free":false,"task":"Your team leads flagged 13 AI-drafted weekly reports, and you noted what was wrong with each. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any report where the lead's own input was wrong. Plain text, under 70 words.","materialLabel":"Your notes on the flagged reports (attached to your prompt automatically)","material":"Field Services org - AI weekly report review, Oct 27-Nov 7 (my notes)\n\n#1 said 42 jobs closed, the tracker says 34\n#2 didn't mention the crane outage that blocked two sites\n#3 still listed the Elm St project as active; it closed two weeks ago\n#4 overtime hours wrong: 120 instead of 210\n#5 left out the safety near-miss on Tuesday\n#6 lead typed the wrong week into the tool; the AI did what it was told\n#7 used last month's headcount (18, we're at 21)\n#8 customer satisfaction 4.8, actual 4.3\n#9 skipped the parts shortage holding up 6 jobs\n#10 referenced a vendor contract we ended in September\n#11 budget used: 62%, real number 71%\n#12 no mention of the late permit for Harbor Rd\n#13 said Dana still leads the north crew; Marco took over last month\n\nContext: you're presenting the fix to the ops director Friday.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits jobs, hours, budget and ratings into separate groups.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes problems without showing which reports belong where.","fix":"Show the shape: \"Type: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting the lead's mistake","why":"The AI counts #6, where the lead gave the tool the wrong week.","fix":"Add: \"Leave out reports where the lead's input was wrong, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out the lead's mistake","patterns":["#6","lead'?s (own )?input|wrong week","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis turns a pile of flags into a short list you can act on. Fix the number of types, ask for ids behind every count, and drop the one that wasn't the AI's fault.","rubricLabels":["Uses these 3 groups (names may differ): wrong numbers #1, #4, #8, #11 = 4; left out a blocker or issue #2, #5, #9, #12 = 4; out-of-date information #3, #7, #10, #13 = 4","Leaves out #6 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}