{"id":"eval-pm-error-analysis-shopping-assistant-traces","audience":"product-managers","category":"evaluation","track":"evals","technique":"format","title":"Group 13 failing shopping-bot traces into 3 failure types","free":false,"task":"You reviewed 13 failing traces from your AI shopping assistant and wrote a quick note on each. Group them into exactly 3 numbered failure categories, each line 'Category name: #ids = count', then one last line 'Total: N'. Leave out any trace that passed on re-check. Plain text, under 70 words.","materialLabel":"Your open-coding notes on each trace (attached to your prompt automatically)","material":"Larkspur Market - AI shopping assistant, failing traces from Nov 1-3\nMy first-pass notes (one per trace):\n\n#1 user asked for nut-free snacks, bot suggested almond granola\n#2 said the blender is in stock, inventory says 0\n#3 user asked \"is this gluten free?\", bot said yes, label says contains wheat\n#4 quoted $39 for the kettle, real price is $49\n#5 recommended a dairy cheese to a user who said they're vegan\n#6 said free shipping on a $20 order, free shipping starts at $35\n#7 told user the toaster ships tomorrow, it's backordered 2 weeks\n#8 RE-CHECKED: actually fine, I misread the trace\n#9 suggested shrimp to a user who said shellfish allergy\n#10 said the 20% coupon works on sale items, it doesn't\n#11 said the rice cooker is in stock in the Elm St store, it isn't\n#12 promised delivery Saturday, store doesn't deliver weekends\n#13 listed a sale price that ended Oct 31\n\nContext: next sprint we can fix 1 category. Our eng lead thinks it's \"probably a retrieval issue\" but hasn't looked at the traces.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many categories","why":"The AI splits the notes into 6 or 7 small groups, so no clear fix stands out.","fix":"Say: \"Exactly 3 categories.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes the problems in prose but you can't see which traces belong where.","fix":"Show the shape: \"Category name: #1, #3 = 2\".","checks":["r2"]},{"title":"Counting the re-checked trace","why":"The AI counts every numbered note, including the one that passed on re-check.","fix":"Add: \"Leave out traces that passed on re-check, and end with Total: N.\"","checks":["r1"]},{"title":"Repeating the guess","why":"The AI may echo 'retrieval issue' instead of grouping what the traces actually show.","fix":"Say: \"Group only by what the notes say.\"","checks":["r0"]},{"title":"Getting markdown back","why":"AI defaults to bold category names and headings.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(categor|group|bucket|type)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Handles the re-checked trace","patterns":["re-?check","#8","exclude","leave out"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis turns trace notes into a short ranked list of problems. Fix the number of categories, ask for the ids behind every count, and drop traces that turned out fine.","rubricLabels":["Uses these 3 groups (names may differ): ignored a stated diet or allergy #1, #3, #5, #9 = 4; wrong price, coupon or shipping-cost info #4, #6, #10, #13 = 4; wrong stock or delivery info #2, #7, #11, #12 = 4","Leaves out #8 and ends with a line saying Total: 12","Each category line lists its # ids followed by the count, and each count equals the number of ids on that line"]}