{"id":"eval-cs-rubric-bot-to-agent-handoff-notes","audience":"customer-support","category":"evaluation","track":"evals","technique":"constraints","title":"Write a 4-line pass/fail rubric for bot handoff notes","free":false,"task":"When your chatbot hands a chat to a human, it writes a handoff note, and agents keep complaining about them. Write exactly 4 numbered criteria, each 'N. Pass if ...; Fail if ...', binary only. They must cover: the order number, what the bot already tried, a mood label from exactly Calm, Upset or Angry, and a 60-word limit. Plain text, under 160 words.","materialLabel":"Sample handoff notes and agent complaints (attached to your prompt automatically)","material":"Pebblecrest Shoes - bot-to-agent handoff notes, samples from Nov 3\n\nNOTE 1\n\"Customer wants to return boots. Seems kind of annoyed. Please help!\"\n\nNOTE 2\n\"Order PC-77120. Customer says the left shoe is a different size. Bot offered an exchange; customer declined and wants a refund. Mood: Upset.\"\n\nNOTE 3\n\"Hello agent! This lovely customer has been chatting with me about her order, which she placed a while ago, and she is wondering whether it might be possible to change the color of the sneakers she ordered to a different one, perhaps navy or maybe gray, and she also mentioned that she might want to change the delivery address too, although she wasn't completely sure about that part yet, so you may want to check with her. Thank you so much!\"\n\nAgent complaints (Slack #support-floor)\n- \"Half the notes don't have the order number, so I have to ask again.\"\n- \"I offer the exchange and the customer says 'the bot already offered that'.\"\n- \"'Seems kind of annoyed' isn't useful. Just tell me calm, upset or angry.\"\n- \"Note 3 is a novel.\"\n- \"The notes are polite at least, I like that.\" (Kofi)\n\nTeam lead: we'll grade 50 notes a week once we have a rubric.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":160}],"mistakes":[{"title":"Getting a rating scale","why":"Asked for a rubric, the AI invents 1-5 levels like 'Excellent' and 'Needs work'.","fix":"Say: \"Binary only: each line is 'Pass if ...; Fail if ...'.\"","checks":["r0","r2"]},{"title":"Not naming the must-haves","why":"The AI writes generic criteria like 'is clear and concise' instead of what agents need.","fix":"List them: \"order number, what the bot tried, mood label, 60-word limit.\"","checks":["r1"]},{"title":"A fuzzy mood check","why":"The AI accepts any feeling word, so 'seems kind of annoyed' still passes.","fix":"Add: \"Mood must be exactly Calm, Upset or Angry.\"","checks":["r1"]},{"title":"Too many criteria","why":"The AI writes 8 or more criteria, including tone and politeness.","fix":"Be exact: \"Exactly 4 numbered criteria, one line each.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for binary pass/fail","patterns":["pass if","pass/fail","binary","no (score|scale)"]},{"label":"Names the must-haves","patterns":["order number","already tried|bot tried","calm|upset|angry","60"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(criteri|line)","exactly \\d+"]},{"label":"Limits each line","patterns":["\\d+\\s*words","one line"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A handoff rubric should test what agents need to pick up the chat. Name the four must-haves, make the mood label an exact list, and ask for binary lines.","rubricLabels":["Every criterion is binary with a concrete condition checkable by reading one note (no 1-5 or 1-10 scales, no points, no 'partially', and no test that is only a vague word like 'clear' or 'helpful')","Covers all four required topics: an order number, what the bot already tried, a mood label that is exactly Calm, Upset or Angry, and 60 words or fewer","Each of the 4 lines has both parts on the same line in the shape 'Pass if ...; Fail if ...', and no criterion is about politeness or tone"]}