Library › AI Evals › Product managers
Group 13 failing shopping-bot traces into 3 failure types
Practice: Asking for a format you can use. About 5 minutes.
About this exercise
You reviewed 13 failing traces from your AI shopping assistant and wrote a quick note on each. Group them into exactly 3 numbered failure categories, each line 'Category name: #ids = count', then one last line 'Total: N'. Leave out any trace that passed on re-check. Plain text, under 70 words.
Common mistakes when you ask AI for this
- Too many categories. The AI splits the notes into 6 or 7 small groups, so no clear fix stands out. Try: Say: "Exactly 3 categories."
- No ids or counts. The AI describes the problems in prose but you can't see which traces belong where. Try: Show the shape: "Category name: #1, #3 = 2".
- Counting the re-checked trace. The AI counts every numbered note, including the one that passed on re-check. Try: Add: "Leave out traces that passed on re-check, and end with Total: N."
- Repeating the guess. The AI may echo 'retrieval issue' instead of grouping what the traces actually show. Try: Say: "Group only by what the notes say."
- Getting markdown back. AI defaults to bold category names and headings. Try: Ask for "plain text, no bold, no headings."
What a strong prompt includes
- Says exactly how many
- Asks for ids and counts
- Handles the re-checked trace
- Asks for a total
- Asks for plain text
Related exercises
- Announce a referral bonus with the steps to claim it (Marketing AI · HR and recruiting)
- Announce a team meeting change people will actually follow (Writing AI · Managers and team leads)
- Answer a 'found an error' question in STAR order (Writing AI · Finance and accounting)
- Answer a wholesale inquiry with clear terms (Writing AI · Small business owners)
- Answer an RFP timeline question in exactly four phases (Writing AI · Sales and customer success)