{"id":"eval-dev-metric-test-generator-catch-rate","audience":"developers","category":"evaluation","track":"evals","technique":"context","title":"Pick the metric that shows if AI-written tests catch bugs","free":false,"task":"Your AI writes unit tests, and the question is whether they catch bugs, not whether they pass. The team wants AI tests to catch bugs at least as often as human tests, and allows at most 3% flaky tests. Write exactly 4 numbered lines: Metric (not pass rate), Threshold (a number and where it comes from), Result (the math, then Ready or Not ready), and Guardrail (the flaky math against 3%). Plain text, under 110 words.","materialLabel":"The test-generator pilot numbers (attached to your prompt automatically)","material":"Quarry Labs - AI unit-test generator pilot (billing-service repo)\n\nGenerated tests: 200\n\nOn clean code\n- 196 of 200 AI tests pass (98% pass rate)\n- 4 of 200 fail on clean code (flaky or wrong)\n\nBug injection (we planted 1 known bug per target function, then re-ran)\n- AI tests that failed because of the planted bug: 60 of 200\n- Human-written tests on the same functions, same planted bugs: 110 of 200 failed (caught the bug)\n\nOther notes\n- Dashboard headline from Jun: \"98% of AI tests pass!\"\n- AI tests took 3 minutes to generate; humans took 2 weeks.\n- Two AI tests only asserted that a function returns something (not None).","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Trusting the pass rate","why":"The AI sees 98% passing and calls the tests great, but passing tests can miss every bug.","fix":"Say: \"Don't use pass rate. Measure how many tests fail when a bug is planted.\"","checks":["r0"]},{"title":"A threshold from nowhere","why":"The AI picks a round number instead of the human baseline in your data.","fix":"Add: \"Base the threshold on the human tests' catch rate.\"","checks":["r1"]},{"title":"No math, no verdict","why":"The AI explains mutation testing but never applies it or says Ready.","fix":"Ask: \"Show the math, then Ready or Not ready, plus the flaky rate vs 3%.\"","checks":["r2"]},{"title":"A testing essay","why":"By default the AI writes sections on coverage and best practices.","fix":"Show the shape: \"1. Metric: 2. Threshold: 3. Result: 4. Guardrail:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Measures bugs caught","patterns":["catch","planted|inject|mutation","fail when"]},{"label":"Rules out pass rate","patterns":["not.{0,20}pass rate","98","pass rate"]},{"label":"Ties the bar to humans","patterns":["human","baseline","110"]},{"label":"Adds the flaky guardrail","patterns":["flaky","3%","guardrail"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A test that never fails proves nothing. Tell the AI what the tests are for, rule out the flattering number, and ask for the math against a human baseline.","rubricLabels":["The metric is the share of tests that fail when a known bug is planted (bug catch rate or mutation kill rate), not pass rate","The threshold is at least 55%, tied to human tests catching 110 of 200","The Result line shows 60 of 200 = 30% and says Not ready, and the Guardrail line shows 4 of 200 = 2% as within the 3% limit"]}