{"id":"eval-pm-metric-bot-resolution-not-deflection","audience":"product-managers","category":"evaluation","track":"evals","technique":"context","title":"Measure real resolution, not deflection, for a support bot","free":false,"task":"Your support bot's dashboard brags about deflection, but leadership wants to know if it actually solves problems. A chat counts as resolved only if the customer doesn't come back about the same issue within 7 days, and leadership's bar is human first-contact resolution plus 5 points. Write exactly 4 numbered lines: Metric (not deflection), Threshold (the number, with the math), Result (the math, then Ready or Not ready), and Guardrail (bot CSAT vs the 4.0 minimum). Plain text, under 110 words.","materialLabel":"The bot's October numbers (attached to your prompt automatically)","material":"Gridley Home Internet - support bot, October\n\nTotal chats started with the bot: 1,000\nChats the bot closed without a human (deflection): 700 (70%)\nOf those 700, customers who came back about the same issue within 7 days: 210\n\nHuman agents (same month, for comparison)\nFirst-contact resolution (no repeat contact within 7 days): 50%\n\nSatisfaction\nBot CSAT: 4.2 / 5\nHuman CSAT: 4.5 / 5\n\nNotes\n- Dashboard headline: \"Bot deflects 70% of chats!\"\n- Leadership is deciding whether to route billing questions to the bot next.\n- Repeat contacts cost about $6 each in agent time.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Trusting the dashboard","why":"The AI repeats 70% deflection, which counts chats where customers came back.","fix":"Say: \"Don't use deflection. A chat is resolved only if there's no repeat within 7 days.\"","checks":["r0"]},{"title":"Missing leadership's bar","why":"The AI uses the human rate as the bar, not knowing leadership wants 5 points better.","fix":"Add: \"Threshold = human first-contact resolution plus 5 points.\"","checks":["r1"]},{"title":"No math, no verdict","why":"The AI explains the metric but never applies it to October.","fix":"Ask: \"Show the math from 1,000 chats, then Ready or Not ready.\"","checks":["r2"]},{"title":"A metrics essay","why":"The AI writes about support KPIs in general.","fix":"Show the shape: \"1. Metric: 2. Threshold: 3. Result: 4. Guardrail:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Rules out deflection","patterns":["not.{0,20}deflect","deflection","repeat"]},{"label":"Defines resolved","patterns":["7 days|seven days","come back|repeat contact"]},{"label":"Ties the bar to humans","patterns":["human","baseline","50"]},{"label":"Adds the CSAT guardrail","patterns":["csat","4\\.0","guardrail"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Deflection counts every chat the bot closed, even when the customer came right back. Define what resolved means, tie the bar to the human team, and ask for the math.","rubricLabels":["The metric is chats resolved without a repeat contact within 7 days, as a share of all chats the bot started (not deflection or closed chats)","The threshold is 55%, shown as human first-contact resolution of 50% plus 5 points","The Result line shows 700 - 210 = 490 of 1,000 = 49% and says Not ready, and the Guardrail line says 4.2 meets the 4.0 minimum"]}