{"id":"eval-mgr-go-no-go-from-eval-results","audience":"managers","category":"evaluation","track":"evals","technique":"verification","title":"Make a launch call from an AI scheduler's eval results","free":false,"task":"You have to decide whether your team's AI meeting scheduler launches Monday. Write exactly 4 numbered lines, one per test category, each 'Category: X of Y = Z% vs bar B% - Pass or Fail', then a last line 'Decision: Launch or Hold - reason'. Use only results from the current prompt version, and note that the ops director raised the time zones bar to 95% yesterday. Plain text, under 110 words.","materialLabel":"The eval results export (attached to your prompt automatically)","material":"Harbor Lane Consulting - \"SlotFinder\" AI scheduler, launch eval (Nov 4)\nCurrent prompt version: v6. Launch bar per category set by the ops director in October.\n\nCategory | prompt version | passed | total | launch bar\nTime zones | v6 | 47 | 50 | 90%\nDouble-booking avoidance | v6 | 50 | 50 | 100%\nRescheduling requests | v5 | 46 | 50 | 85%\nRescheduling requests | v6 | 40 | 50 | 85%\nHoliday and out-of-office | v6 | 45 | 50 | 90%\n\nNotes\n- v5 rescheduling results are from two weeks ago, before the tone changes in v6.\n- Product lead Simone: \"Overall we're at 93%, that's an A!\"\n- Most client complaints last quarter were about rescheduling.\n- Engineering can do one more fix sprint before the holiday freeze if needed.\n- Launch slot: Monday 9am.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Using the old result","why":"The table has two rescheduling rows, and the AI may pick the better-looking v5 one.","fix":"Say: \"Use only v6 results.\"","checks":["r0"]},{"title":"Averaging it all together","why":"The AI repeats the 93% overall number, hiding the one category that fails.","fix":"Add: \"Judge each category against its own bar.\"","checks":["r2"]},{"title":"Hiding the math","why":"The AI says 'rescheduling is a bit low' without numbers.","fix":"Show the shape: \"Category: X of Y = Z% vs bar B% - Pass or Fail.\"","checks":["r1"]},{"title":"Using the old bar","why":"The table still shows 90% for time zones, so the AI passes a category that now fails.","fix":"Tell it what changed: \"The time zones bar is now 95%.\"","checks":["r1","r2"]},{"title":"A long memo","why":"The AI writes a launch memo with risks and next steps.","fix":"Be exact: \"4 numbered lines, then one Decision line.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Uses only the current version","patterns":["v6","current (prompt|version)","not v5","latest"]},{"label":"Compares each category to its bar","patterns":["bar","threshold","each category","per category"]},{"label":"Asks for the math","patterns":["x of y","math","of 50","show"]},{"label":"Asks for a clear decision","patterns":["launch or hold","decision","go.{0,5}no.?go"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"One average can hide the category that will hurt clients. Ask the AI to check each category against its own bar, with the math, using only the current version.","rubricLabels":["Uses the v6 rescheduling result (40 of 50 = 80%), not the v5 one (46 of 50)","Shows the math and verdict per category against the current bars: time zones 47 of 50 = 94% vs 95% Fail, double-booking 50 of 50 = 100% Pass, rescheduling 80% vs 85% Fail, holiday 45 of 50 = 90% Pass","The last line says Hold and names both failing categories (rescheduling and time zones), and does not use an overall average to decide"]}