Library › AI Evals › Software developers

Pick the metric that shows if AI-written tests catch bugs

Practice: Giving the right context. About 5 minutes.

About this exercise

Your AI writes unit tests, and the question is whether they catch bugs, not whether they pass. The team wants AI tests to catch bugs at least as often as human tests, and allows at most 3% flaky tests. Write exactly 4 numbered lines: Metric (not pass rate), Threshold (a number and where it comes from), Result (the math, then Ready or Not ready), and Guardrail (the flaky math against 3%). Plain text, under 110 words.

Common mistakes when you ask AI for this

  • Trusting the pass rate. The AI sees 98% passing and calls the tests great, but passing tests can miss every bug. Try: Say: "Don't use pass rate. Measure how many tests fail when a bug is planted."
  • A threshold from nowhere. The AI picks a round number instead of the human baseline in your data. Try: Add: "Base the threshold on the human tests' catch rate."
  • No math, no verdict. The AI explains mutation testing but never applies it or says Ready. Try: Ask: "Show the math, then Ready or Not ready, plus the flaky rate vs 3%."
  • A testing essay. By default the AI writes sections on coverage and best practices. Try: Show the shape: "1. Metric: 2. Threshold: 3. Result: 4. Guardrail:"

What a strong prompt includes

  • Measures bugs caught
  • Rules out pass rate
  • Ties the bar to humans
  • Adds the flaky guardrail
  • Asks for plain text

Practice this now ↑

Related exercises