Library › AI Evals › Managers and team leads
Make a launch call from an AI scheduler's eval results
Practice: Catching mistakes and made-up facts. About 5 minutes.
About this exercise
You have to decide whether your team's AI meeting scheduler launches Monday. Write exactly 4 numbered lines, one per test category, each 'Category: X of Y = Z% vs bar B% - Pass or Fail', then a last line 'Decision: Launch or Hold - reason'. Use only results from the current prompt version, and note that the ops director raised the time zones bar to 95% yesterday. Plain text, under 110 words.
Common mistakes when you ask AI for this
- Using the old result. The table has two rescheduling rows, and the AI may pick the better-looking v5 one. Try: Say: "Use only v6 results."
- Averaging it all together. The AI repeats the 93% overall number, hiding the one category that fails. Try: Add: "Judge each category against its own bar."
- Hiding the math. The AI says 'rescheduling is a bit low' without numbers. Try: Show the shape: "Category: X of Y = Z% vs bar B% - Pass or Fail."
- Using the old bar. The table still shows 90% for time zones, so the AI passes a category that now fails. Try: Tell it what changed: "The time zones bar is now 95%."
- A long memo. The AI writes a launch memo with risks and next steps. Try: Be exact: "4 numbered lines, then one Decision line."
What a strong prompt includes
- Uses only the current version
- Compares each category to its bar
- Asks for the math
- Asks for a clear decision
- Asks for plain text
Related exercises
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Approve or return expense claims against the policy (Productivity AI · Finance and accounting)
- Approve part of an overtime request with the math (Writing AI · Managers and team leads)
- Catch a chatbot's mistake on your invoice total (Data AI · Freelancers and creators)
- Catch a chatbot's mistakes about a supplier's return policy (Research & AI Search · Small business owners)