Library › AI Evals › Product managers

Pick the right metric and bar for an urgent-ticket tagger

Practice: Giving the right context. About 5 minutes.

About this exercise

Your AI tags support tickets, and the costly failure is missing an urgent one. On-call agreed to accept false alarms on up to 3% of Normal tickets. Write exactly 4 numbered lines: Metric (what you measure, not overall accuracy), Threshold (a number and where it comes from in the data), Pilot result (the math, then Ready or Not ready), and Guardrail (the false-alarm math against the 3% limit). Plain text, under 110 words.

Common mistakes when you ask AI for this

  • Trusting the headline number. The AI sees 97% accuracy and calls it great, even though almost every ticket is Normal. Try: Say: "Don't use overall accuracy. Measure how many urgent tickets it catches."
  • A threshold from nowhere. The AI picks a round number like 90% without tying it to anything in your data. Try: Add: "Base the threshold on the human baseline in the data."
  • No math, no verdict. The AI explains the metric but never applies it to the pilot or says if it's ready. Try: Ask: "Show the pilot math, then say Ready or Not ready."
  • No guardrail. The AI doesn't know on-call's 3% false-alarm limit, so it never checks the cost of over-flagging. Try: Add: "Guardrail: false alarms out of Normal tickets vs the 3% limit."
  • A metrics essay. By default the AI explains precision, recall and F1 with headings. Try: Show the shape: "1. Metric: ... 2. Threshold: ... 3. Pilot result: ... 4. Guardrail: ..."

What a strong prompt includes

  • Focuses on missed urgent tickets
  • Rules out overall accuracy
  • Ties the bar to the baseline
  • Asks for the math and verdict
  • Asks for plain text

Practice this now ↑

Related exercises