{"id":"eval-cs-error-analysis-low-csat-chats","audience":"customer-support","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 low-rated bot chats into 3 failure types","free":false,"task":"You read 13 bot chats that customers rated 1 star and noted what went wrong in each. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any chat where the low rating wasn't about the bot. Plain text, under 70 words.","materialLabel":"Your notes on the 1-star chats (attached to your prompt automatically)","material":"Brightloop Electronics - 1-star bot chats, Nov 1-5, my notes\n\n#1 customer asked for a human 3 times, bot kept answering itself\n#2 bot said order would arrive Tuesday, tracking says next week\n#3 bot looped: asked for order number, then asked again after getting it\n#4 customer typed \"agent please\", bot replied with an FAQ link\n#5 bot said the 2-year warranty covers drops, it doesn't\n#6 rating was about the courier being rude at the door, bot chat was fine\n#7 bot asked the same \"what's your email?\" question 3 times\n#8 customer said \"let me talk to someone\", no handoff offered\n#9 bot quoted the old $5 restocking fee, it's free now\n#10 bot restarted the conversation after the customer uploaded a photo\n#11 \"human!!!\" - bot said it didn't understand\n#12 bot said store pickup is available in Denver, that store closed\n#13 bot kept asking to confirm the address after it was confirmed\n\nContext: support director wants one fix for next sprint.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits wrong dates, wrong fees and wrong stores into separate groups.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes problems without showing which chats belong where.","fix":"Show the shape: \"Type: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting a non-bot rating","why":"The AI counts every 1-star chat, including the one about the courier.","fix":"Add: \"Leave out chats where the rating wasn't about the bot, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out the non-bot chat","patterns":["#6","courier","not about the bot","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Low ratings have different causes, and not all of them are the bot. Fix the number of types, ask for ids behind every count, and leave out ratings about something else.","rubricLabels":["Uses these 3 groups (names may differ): didn't hand off to a human when asked #1, #4, #8, #11 = 4; wrong information #2, #5, #9, #12 = 4; looped or repeated questions #3, #7, #10, #13 = 4","Leaves out #6 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}