{"id":"eval-pm-error-analysis-trip-planner-complaints","audience":"product-managers","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 trip-planner complaints into 3 failure types","free":false,"task":"You read 13 user complaints about your AI trip planner's itineraries and noted the real problem in each. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any complaint that turned out to be about the airline, not the planner. Plain text, under 70 words.","materialLabel":"Your notes on the complaints (attached to your prompt automatically)","material":"Wayloft - AI trip planner complaints, Nov 1-6 (my notes on each)\n\n#1 planned a museum visit on a Monday, the museum is closed Mondays\n#2 two activities 40 km apart scheduled 15 minutes apart\n#3 suggested a restaurant that closed in 2024\n#4 day 3 had 11 hours of activities with no meal break\n#5 user's flight was cancelled by the airline; planner did nothing wrong\n#6 hotel check-in planned for 9am, hotel says 3pm\n#7 a 2-hour hike squeezed between lunch at 12 and a tour at 1\n#8 recommended a ferry route that stopped running last year\n#9 night market planned for Tuesday, it only runs Fri-Sun\n#10 a 6-hour day trip on the same day as the late-night arrival\n#11 listed a \"beach bar\" that is now a parking lot\n#12 three cities in one day for a family with toddlers\n#13 booked-out tour listed as available, tour company stopped it in 2025\n\nContext: one fix next sprint. Eng thinks the model needs fresher place data.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits hotels, museums and markets into separate groups, so no fix stands out.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes problems without showing which complaints belong where.","fix":"Show the shape: \"Type: #1, #6 = 2\".","checks":["r2"]},{"title":"Counting the airline's mistake","why":"The AI counts #5, which wasn't the planner's fault.","fix":"Add: \"Leave out complaints about the airline, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out the airline complaint","patterns":["#5","airline","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis shows which failure is most common before you pick a fix. Fix the number of types, ask for the ids behind every count, and drop complaints the planner couldn't have prevented.","rubricLabels":["Uses these 3 groups (names may differ): wrong opening days or hours #1, #6, #9 = 3; impossible or overpacked timing #2, #4, #7, #10, #12 = 5; place or service no longer exists #3, #8, #11, #13 = 4","Leaves out #5 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}