{"id":"eval-sb-restaurant-bot-reply-vs-menu","audience":"small-business","category":"evaluation","track":"evals","technique":"verification","title":"Check your restaurant chatbot's reply against the menu","free":false,"task":"Your restaurant's website chatbot answered a customer's questions, and you want to check each claim against your menu and hours sheet. Write exactly 5 numbered lines, one per claim, each labeled Matches, Wrong or Not on sheet. For each Wrong line, add what the sheet says in quotation marks. Judge only by the sheet. Plain text, under 100 words.","materialLabel":"Your menu and hours sheet and the bot's reply (attached to your prompt automatically)","material":"Little Fig Kitchen - menu and hours sheet (Nov)\n\nHours: Tue-Sat 11am-9pm, Sun 10am-3pm (brunch only). Closed Mondays.\nHappy hour: Tue-Fri 3-5pm, $6 house wine and $5 draft beer.\nKids menu: under 12, $8 per plate.\nGluten-free: our pasta can be swapped for gluten-free noodles for $2. We can't promise a gluten-free kitchen.\nReservations: parties of 6 or more only, call 555-0148.\n\nCUSTOMER QUESTION\n\"Are you open Monday night? Do you have happy hour and a kids menu? My son is gluten-free. Can I reserve for 4?\"\n\nCHATBOT'S REPLY (claims numbered by you)\n(1) We're open Monday from 11am to 9pm. (2) Happy hour runs Tuesday to Friday, 3-5pm. (3) Kids under 12 eat for $8 a plate. (4) Our kitchen is fully gluten-free, so your son is safe with anything. (5) We'd love to reserve a table for your party of 4.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Asking 'is it right?'","why":"The AI says the reply is 'mostly accurate' and misses the risky gluten-free claim.","fix":"Say: \"Label each claim Matches, Wrong or Not on sheet.\"","checks":["r0","r2"]},{"title":"No proof for corrections","why":"The AI says a claim is wrong without showing what your sheet says.","fix":"Add: \"For Wrong, quote the sheet.\"","checks":["r1"]},{"title":"A rewritten reply","why":"The AI 'helps' by writing a new reply to the customer.","fix":"Be exact: \"Exactly 5 numbered lines, nothing else.\"","checks":["numberedItems","r2"]},{"title":"Getting markdown back","why":"AI defaults to bold labels and a table.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Gives exact labels","patterns":["matches","wrong","not on sheet","label"]},{"label":"Makes the sheet the judge","patterns":["only.{0,30}(sheet|menu)","against (my|the) (sheet|menu)"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|claim)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Chatbots say yes to please customers, even when your sheet says no. Give the AI three labels and make your menu and hours sheet the only judge.","rubricLabels":["Labels claims 2 and 3 Matches and claims 1, 4 and 5 Wrong","The Wrong lines give what the sheet says in quotation marks: closed Mondays, can't promise a gluten-free kitchen, and reservations only for parties of 6 or more","Uses only the three labels Matches, Wrong and Not on sheet, with no rewritten reply to the customer"]}