{"id":"eval-sb-golden-answer-custom-cake-faq","audience":"small-business","category":"evaluation","track":"evals","technique":"context","title":"Write the golden answer your bakery chatbot is graded on","free":false,"task":"You're testing your bakery's website chatbot and need a golden answer for one test question. Since Dec 1, custom cakes need 5 days' notice instead of 72 hours. Write exactly 3 numbered lines: '1. Answer: ...' (a direct yes or no), '2. Why: ...' (the rule and the dates), and '3. Next step: ...' (what the customer can do instead). Under 70 words, plain text.","materialLabel":"The test question and your FAQ page (attached to your prompt automatically)","material":"TEST QUESTION #7 for the Sweet Hollow Bakery chatbot\nAsked on: Wednesday, Dec 10\nCustomer: \"Hi! Can I order a custom birthday cake with my son's name on it for pickup this Saturday, Dec 13?\"\n\nFAQ PAGE (sweethollowbakery.example/faq, last edited in June)\n- Custom cakes: we need 72 hours' notice. Prices start at $45 for a 6-inch cake.\n- Order custom cakes at sweethollowbakery.example/custom or by phone.\n- Our display case has ready-made cakes daily, including chocolate and vanilla. We can pipe a name on any display cake while you wait, free.\n- Hours: Tues-Sat 7am-4pm. Closed Sun-Mon.\n- We can't guarantee nut-free cakes; our kitchen uses nuts.\n\nOwner notes\n- December is our busiest month.\n- The chatbot's answers in testing were often 120+ words with lots of exclamation points.\n- Golden answers should be short: they're what we grade the chatbot against.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Using the old rule","why":"The FAQ still says 72 hours, which would make Saturday look fine.","fix":"Tell it: \"Since Dec 1, custom cakes need 5 days' notice.\"","checks":["r0","r1"]},{"title":"A no with no way forward","why":"The AI says no and stops, so the customer leaves.","fix":"Ask: \"End with what they can do instead, from the FAQ.\"","checks":["r2"]},{"title":"A long cheerful reply","why":"The AI writes a friendly paragraph with exclamation points, which is hard to grade against.","fix":"Show the shape: \"1. Answer: 2. Why: 3. Next step:\"","checks":["numberedItems","maxWords"]},{"title":"Getting markdown back","why":"AI likes bold labels, which show up as symbols in your test sheet.","fix":"Ask for \"plain text, no bold.\"","checks":["plainText"]}],"promptHints":[{"label":"Gives the new rule","patterns":["5 days|five days","dec(ember)? 1","72"]},{"label":"Shows the three-line shape","patterns":["answer:","why:","next step"]},{"label":"Asks for an alternative","patterns":["instead","alternative","display","next step"]},{"label":"Keeps to the FAQ","patterns":["only.{0,30}(faq|page|material)","don't (add|invent)","nothing.{0,10}not in"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer is the short, correct reply you'll grade the chatbot against. Tell the AI the rule that changed, show the exact shape, and ask for a next step from the FAQ.","rubricLabels":["Line 1 says no, a custom cake can't be ready for Saturday, Dec 13","Line 2 gives the 5-day notice rule (not 72 hours) and the dates, showing Dec 10 to Dec 13 is only 3 days","Line 3 offers a display-case cake with the name piped on for free, and the answer adds nothing that isn't in the FAQ (no new prices, flavors or hours)"]}