{"id":"eval-cs-pushy-bot-feedback-to-checks","audience":"customer-support","category":"evaluation","track":"evals","technique":"verification","title":"Turn \"the bot feels pushy\" into 4 testable checks","free":false,"task":"Customers keep telling agents your cancellation bot 'feels pushy', and you need checks a QA reviewer can apply to one chat. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one chat] - e.g. \"quote from the feedback\"'. Don't use vague words like pushy, aggressive or salesy as the test, and skip praise and unrelated complaints. Plain text, under 130 words.","materialLabel":"Customer feedback collected by agents (attached to your prompt automatically)","material":"Gridley Home Internet - feedback about the cancellation bot (agents' notes, Oct)\n\n- \"It offered me a discount four times before letting me cancel.\"\n- \"I said 'cancel' and it asked 'Are you sure?' three times.\"\n- \"It never told me how to actually cancel, just kept showing deals.\"\n- \"It said 'Most customers who leave regret it.' Felt like a guilt trip.\"\n- \"To be fair, the first discount was a good deal, I almost took it.\"\n- \"The install tech was late last month.\" (agent note: unrelated)\n- \"Took 12 messages to get a cancellation number.\"\n- \"It felt so pushy.\"\n- \"It said the offer 'expires in 5 minutes'. Is that real?\"\n\nContext: retention team wants to keep one save offer. Legal wants the cancel path to be easy. You need checks for grading 100 chats next week.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":130}],"mistakes":[{"title":"Keeping the vague words","why":"The AI rewrites 'pushy' as 'respects the customer's choice', which is just as hard to grade.","fix":"Say: \"Each check must be something you can see or count in one chat.\"","checks":["r0"]},{"title":"No example from real data","why":"The AI writes generic retention tips instead of tying checks to what customers said.","fix":"Add: \"Quote an example from the feedback on each line.\"","checks":["r1"]},{"title":"Banning all offers","why":"The AI fails any chat with an offer, but retention wants to keep one.","fix":"Add: \"One save offer is fine. Skip praise and unrelated complaints.\"","checks":["r2"]},{"title":"Too many checks","why":"The AI turns every comment into a check.","fix":"Be exact: \"Exactly 4 numbered checks.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for observable checks","patterns":["see or count","observable","countable","testable"]},{"label":"Asks for quoted examples","patterns":["quote","example","e\\.g\\."]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(check|line|criteri)","exactly \\d+"]},{"label":"Allows one offer","patterns":["one (save )?offer","first discount","praise","unrelated"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"'Pushy' can't be graded, but 'offered a discount more than once' can. Ask the AI to turn each complaint into something you can count in one chat, with a real quote.","rubricLabels":["Each check's test is something visible or countable in one chat (more than one discount offer, repeated 'Are you sure?' prompts, no cancel steps given, guilt or urgency lines like 'expires in 5 minutes', too many messages before a cancellation number), not a judgment word like pushy, aggressive or salesy","Each of the 4 lines includes an example quoted from the feedback, such as \"Most customers who leave regret it\" or \"Are you sure?\"","No check fails a chat for making one save offer, and no check is about the install tech"]}