{"id":"evals3-regression-hiding-in-other-column","audience":"analysts","category":"evaluation","track":"evals","stage":"fix-prompt","level":"advanced","technique":"verification","title":"Find the regression hiding behind a rising overall score","free":false,"task":"Spokewise patched Pedalbot to say 'not covered' when the policy is silent, then reran 50 test cases. The overall score went up, so the PM wants to ship. Write a plain text message to Dana in exactly 3 lines: 'Improved:', 'Regressed:' and 'Call:'. Write before and after exactly like 36 of 40 = 90% -> 32 of 40 = 80%. Improved is the column that got better; Regressed is the column that got worse plus what the bot now does wrong; Call starts with 'Call: Hold' or 'Call: Ship' and gives a reason. Under 85 words.","materialLabel":"Before and after results (attached to your prompt automatically)","material":"SPOKEWISE - PEDALBOT PATCH v5 -> v6 (same 50 test cases both runs)\nThe patch added: \"If the policy does not answer the question, reply that it is not covered.\"\n\nColumn | cases | v5 passed | v6 passed\nPolicy answers the question: reply is correct | 40 | 36 | 32\nPolicy is silent: reply says \"not covered\" | 10 | 4 | 10\nOverall (all 50 cases) | 50 | 40 | 42\nReply under 60 words | 50 | 44 | 44\n\nSAMPLE OF THE 4 NEW MISSES (policy answers the question)\n- \"Is the helmet free?\" v6: \"That is not covered.\" (policy: free on weekly rentals)\n- \"Cost of a lost key?\" v6: \"That is not covered.\" (policy: $45)\n- \"Can I bring the bike on the ferry?\" v6: \"That is not covered.\" (policy: yes, with a $2 ticket)\n- \"Late fee for an e-bike?\" v6: \"That is not covered.\" (policy: $8 per started hour)\n\nNOTES\n- Overall: 80% to 84%, so Sam (PM): \"It went up, ship it.\"\n- Customers ask about covered topics far more often than uncovered ones.\n- Dana owns the go-live decision.","rules":[{"type":"plainText"},{"type":"maxWords","max":85}],"mistakes":[{"title":"Cheering the overall score","why":"The AI repeats 80% to 84% and says the patch worked.","fix":"Say: \"Check every column, not just the overall score.\"","checks":["r2"]},{"title":"Missing the hidden drop","why":"The AI reports only the column the patch targeted.","fix":"Add: \"Find any column that got worse.\"","checks":["r1"]},{"title":"Percent without counts","why":"The AI gives 'up 60 points' with no counts.","fix":"Say: \"Format: X of Y = Z% before and after.\"","checks":["r0"]},{"title":"Not saying what went wrong","why":"The AI says a column dropped but not what the bot now does.","fix":"Say: \"Say what the bot now does wrong.\"","checks":["r1"]},{"title":"Long and formatted","why":"AI adds bold headings and a summary.","fix":"Ask for \"3 lines, plain text, under 85 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Asks to check every column","patterns":["every column","each column","other column","not just"]},{"label":"Asks for before and after math","patterns":["before.{0,20}after","x of y","of 40"]},{"label":"Asks what got worse","patterns":["regress","got worse","dropped"]},{"label":"Asks for a ship or hold call","patterns":["ship or hold","call:","hold"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A patch can fix what you aimed at and quietly break something else. Read every column before and after, and don't let a rising overall score settle the question.","mission":"Report what the patch fixed and what it broke.","musts":["Three labeled lines","Math on both columns","Call says hold"],"rubricLabels":["The Improved line reports the silent-policy column formatted exactly as 4 of 10 = 40% -> 10 of 10 = 100%","The Regressed line reports the answerable column formatted exactly as 36 of 40 = 90% -> 32 of 40 = 80%, and says the bot now wrongly says not covered for questions the policy answers","The Call line starts with \"Call: Hold\" and gives the reason that the overall rise from 80% to 84% hides the drop in answerable questions"]}