{"id":"eval-off-itinerary-vs-booking-confirmations","audience":"office","category":"evaluation","track":"evals","technique":"verification","title":"Check an AI travel itinerary against the bookings","free":false,"task":"An AI built your manager's trip itinerary from the booking emails, and you want to catch mistakes before you send it. Write exactly 6 numbered lines, one per itinerary line, each labeled Matches, Wrong or Not in bookings. For each Wrong line, add the booking's version in quotation marks. Judge only by the confirmations. Plain text, under 110 words.","materialLabel":"The booking confirmations and the AI itinerary (attached to your prompt automatically)","material":"BOOKING CONFIRMATIONS (forwarded to you, trip for Helen Marsh, Nov 17-19)\n\nFlight - Skyline Air SK 482, Mon Nov 17, departs Columbus 7:10am, arrives Denver 8:55am (local). Seat 14C. Conf. QX7RT2.\nHotel - Larimer Grand, check-in Mon Nov 17, check-out Wed Nov 19. 2 nights. King room. Conf. 448190.\nCar - none booked (Helen said she'll use rideshare).\nReturn flight - Skyline Air SK 517, Wed Nov 19, departs Denver 5:40pm, arrives Columbus 10:50pm.\nDinner - Copper Table, Tue Nov 18, 7:00pm, party of 4, under \"Marsh\".\n\nAI ITINERARY (draft)\n1. Mon Nov 17: Fly Columbus to Denver on SK 482, departs 7:10am, seat 14C.\n2. Mon Nov 17: Pick up rental car at Denver airport.\n3. Mon-Wed: Larimer Grand, 3 nights, king room.\n4. Tue Nov 18: Dinner at Copper Table, 7:00pm, party of 4.\n5. Wed Nov 19: Breakfast meeting at hotel lobby, 8am.\n6. Wed Nov 19: Fly home on SK 517, departs 5:40pm.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Asking 'is it right?'","why":"The AI says the itinerary 'looks good' and skims past the extra night and the made-up meeting.","fix":"Say: \"Label each line Matches, Wrong or Not in bookings.\"","checks":["r0","r2"]},{"title":"No proof for corrections","why":"The AI says a line is wrong without showing what the booking says.","fix":"Add: \"For Wrong, quote the booking in quotation marks.\"","checks":["r1"]},{"title":"Getting a new itinerary","why":"The AI 'fixes' it by rewriting the whole thing, so you can't see what changed.","fix":"Say: \"Exactly 6 numbered lines, one per itinerary line, nothing else.\"","checks":["numberedItems","r2"]},{"title":"Getting markdown back","why":"AI defaults to a bold table that won't paste into your email.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Gives exact labels","patterns":["matches","wrong","not in bookings","label"]},{"label":"Makes the bookings the judge","patterns":["only.{0,30}(booking|confirmation)","against the (booking|confirmation)"]},{"label":"Asks for a quote","patterns":["quote","what the booking says"]},{"label":"Says exactly how many","patterns":["\\b(6|six)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI itineraries often add an extra night or a meeting nobody booked. Give the AI three labels and make the confirmations the only judge.","rubricLabels":["Labels lines 1, 4 and 6 Matches, line 3 Wrong, and line 5 Not in bookings","Line 2 (the rental car) is labeled Wrong or Not in bookings, and line 3 quotes the booking's 2 nights","Uses only the three labels Matches, Wrong and Not in bookings, with no rewritten itinerary, packing tips or intro"]}