{"id":"eval-off-error-analysis-room-booking-assistant","audience":"office","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 bad AI room bookings into 3 failure types","free":false,"task":"Your office's AI room-booking assistant made 13 bad bookings last week, and you noted what went wrong with each. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any booking where the person typed the wrong details themselves. Plain text, under 70 words.","materialLabel":"Your notes on the bad bookings (attached to your prompt automatically)","material":"Calder & Webb (office admin) - AI room bookings, problems Nov 3-7\n\n#1 booked Room 4 for 12 people, it seats 6\n#2 booked 2pm instead of 2:30pm\n#3 booked the Oak Room, which is closed for painting all week\n#4 \"next Tuesday\" booked for this Tuesday\n#5 booked the phone booth for a 5-person meeting\n#6 requester typed the wrong date in their own request, assistant did what they asked\n#7 booked the 3rd floor kitchen as a meeting room (not bookable)\n#8 booked 9am Central for a request that said 9am Eastern\n#9 booked a 4-seat room for a 10-person training\n#10 booked Room 2 during its weekly cleaning block\n#11 meeting set for 30 minutes, request said 1 hour\n#12 put an 8-person interview panel in a 4-seat room\n#13 booked the boardroom, which is reserved for the partners on Fridays\n\nContext: the vendor asked which problem to fix first.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits wrong dates, wrong times and wrong time zones into separate groups.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes problems without showing which bookings belong where.","fix":"Show the shape: \"Type: #1, #5 = 2\".","checks":["r2"]},{"title":"Counting a user mistake","why":"The AI counts #6, where the person typed the wrong date themselves.","fix":"Add: \"Leave out bookings where the requester made the mistake, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out the user mistake","patterns":["#6","requester|typed the wrong|their own","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"To tell a vendor what to fix first, you need a short list of failure types with counts. Fix the number of types, ask for the ids behind each count, and drop the one that wasn't the AI's fault.","rubricLabels":["Uses these 3 groups (names may differ): room too small for the group #1, #5, #9, #12 = 4; wrong date, time or length #2, #4, #8, #11 = 4; room unavailable or not bookable #3, #7, #10, #13 = 4","Leaves out #6 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}