Library › AI Evals › Product managers

Measure real resolution, not deflection, for a support bot

Practice: Giving the right context. About 5 minutes.

About this exercise

Your support bot's dashboard brags about deflection, but leadership wants to know if it actually solves problems. A chat counts as resolved only if the customer doesn't come back about the same issue within 7 days, and leadership's bar is human first-contact resolution plus 5 points. Write exactly 4 numbered lines: Metric (not deflection), Threshold (the number, with the math), Result (the math, then Ready or Not ready), and Guardrail (bot CSAT vs the 4.0 minimum). Plain text, under 110 words.

Common mistakes when you ask AI for this

  • Trusting the dashboard. The AI repeats 70% deflection, which counts chats where customers came back. Try: Say: "Don't use deflection. A chat is resolved only if there's no repeat within 7 days."
  • Missing leadership's bar. The AI uses the human rate as the bar, not knowing leadership wants 5 points better. Try: Add: "Threshold = human first-contact resolution plus 5 points."
  • No math, no verdict. The AI explains the metric but never applies it to October. Try: Ask: "Show the math from 1,000 chats, then Ready or Not ready."
  • A metrics essay. The AI writes about support KPIs in general. Try: Show the shape: "1. Metric: 2. Threshold: 3. Result: 4. Guardrail:"

What a strong prompt includes

  • Rules out deflection
  • Defines resolved
  • Ties the bar to humans
  • Adds the CSAT guardrail
  • Asks for plain text

Practice this now ↑

Related exercises