{"id":"eval-pm-metric-threshold-urgent-ticket-tagger","audience":"product-managers","category":"evaluation","track":"evals","technique":"context","title":"Pick the right metric and bar for an urgent-ticket tagger","free":false,"task":"Your AI tags support tickets, and the costly failure is missing an urgent one. On-call agreed to accept false alarms on up to 3% of Normal tickets. Write exactly 4 numbered lines: Metric (what you measure, not overall accuracy), Threshold (a number and where it comes from in the data), Pilot result (the math, then Ready or Not ready), and Guardrail (the false-alarm math against the 3% limit). Plain text, under 110 words.","materialLabel":"Pilot results and team notes (attached to your prompt automatically)","material":"Quillfire Support - AI ticket tagger pilot (October)\n\nThe tagger labels each new ticket Urgent or Normal. Urgent tickets (outages, security, payments failing) must reach on-call within 15 minutes; Normal can wait a day.\n\nPilot numbers (2,000 tickets)\n- Overall accuracy: 97.5% (1,950 of 2,000 tagged the same as the human reviewer)\n- Tickets humans marked Urgent: 100\n- Of those 100, the AI also tagged Urgent: 88\n- Tickets humans marked Normal: 1,900\n- Of those, the AI tagged Urgent (false alarms): 38\n\nBaseline\n- Our human triage team catches 95 of every 100 urgent tickets (Q3 audit).\n\nFrom the incident review\n- One missed urgent ticket in Q3 was a payments outage for a 900-seat customer. It took 6 hours to reach on-call.\n- On-call says: \"Extra false alarms are annoying, but a missed outage is a disaster. We'd rather over-flag.\"\n\nExec slide draft (from Jordan): \"97.5% accurate - ready to launch!\"\n\nOther notes\n- Normal tickets mislabeled as Urgent cost about 2 minutes of on-call time each.\n- Launch review is next Wednesday.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Trusting the headline number","why":"The AI sees 97% accuracy and calls it great, even though almost every ticket is Normal.","fix":"Say: \"Don't use overall accuracy. Measure how many urgent tickets it catches.\"","checks":["r0"]},{"title":"A threshold from nowhere","why":"The AI picks a round number like 90% without tying it to anything in your data.","fix":"Add: \"Base the threshold on the human baseline in the data.\"","checks":["r1"]},{"title":"No math, no verdict","why":"The AI explains the metric but never applies it to the pilot or says if it's ready.","fix":"Ask: \"Show the pilot math, then say Ready or Not ready.\"","checks":["r2"]},{"title":"No guardrail","why":"The AI doesn't know on-call's 3% false-alarm limit, so it never checks the cost of over-flagging.","fix":"Add: \"Guardrail: false alarms out of Normal tickets vs the 3% limit.\"","checks":["r2"]},{"title":"A metrics essay","why":"By default the AI explains precision, recall and F1 with headings.","fix":"Show the shape: \"1. Metric: ... 2. Threshold: ... 3. Pilot result: ... 4. Guardrail: ...\"","checks":["numberedItems","plainText","maxWords"]}],"promptHints":[{"label":"Focuses on missed urgent tickets","patterns":["urgent","miss","recall","catch"]},{"label":"Rules out overall accuracy","patterns":["not.{0,20}accuracy","accuracy.{0,20}mislead","97"]},{"label":"Ties the bar to the baseline","patterns":["baseline","human","95"]},{"label":"Asks for the math and verdict","patterns":["math","ready","show"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A metric should measure the failure that hurts most. Tell the AI which mistake is costly, rule out the flattering number, and ask it to apply the bar to the pilot.","rubricLabels":["The metric is the share of human-marked urgent tickets the AI also tags Urgent (recall or catch rate on urgent), not overall accuracy or precision","The threshold is at least 95% and is tied to the human team catching 95 of 100 urgent tickets","The pilot result line shows 88 of 100 = 88% and says Not ready, and the Guardrail line shows 38 of 1,900 = 2% as within the 3% limit"]}