{"id":"eval-dev-error-analysis-codegen-test-failures","audience":"developers","category":"evaluation","track":"evals","technique":"format","title":"Group 13 failing code-assistant runs into 3 causes","free":false,"task":"You reviewed 13 failing runs from your code assistant's eval suite and left a note on each. Group them into exactly 3 numbered failure categories, each line 'Category: #ids = count', then one last line 'Total: N'. Leave out any run that passed when re-run (flaky). Plain text, under 70 words.","materialLabel":"Your notes on the failing runs (attached to your prompt automatically)","material":"codehelper eval suite - nightly run Nov 4, 13 failures, my notes\n\n#1 imported pandas, which isn't installed in the sandbox\n#2 called df.append(), removed in the pandas version we pin\n#3 test expects a list, function returns a generator\n#4 used requests.get without the requests package available\n#5 re-ran it, passed. flaky network mock\n#6 used Python 3.12 syntax (type statement), runtime is 3.10\n#7 returned None instead of raising ValueError on bad input\n#8 imported numpy, not in requirements.txt\n#9 used str.removeprefix wrongly... actually fine, it's 3.9+, my bad. real issue: it returns \"\" instead of None for empty input, test expects None\n#10 called os.getenv on a key the test never sets, then crashed on None + str\n#11 import of a module that doesn't exist (\"fastjsonx\")\n#12 sorted descending, spec says ascending\n#13 used match statement correctly but returned dict where test wants tuple\n\nContext: we can fix one category this sprint, probably by changing the system prompt or the sandbox. Tech lead thinks \"the model is bad at Python\".","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many categories","why":"The AI makes a group per error type, so nothing points to one fix.","fix":"Say: \"Exactly 3 categories.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI explains the errors but you can't see which runs belong where.","fix":"Show the shape: \"Category: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting the flaky run","why":"The AI counts every note, including the one that passed on re-run.","fix":"Add: \"Leave out runs that passed on re-run, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headers and code formatting.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(categor|group|bucket|cause)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Handles the flaky run","patterns":["flaky","re-?run","#5","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis tells you which fix removes the most failures. Fix the number of categories, ask for the ids behind every count, and drop the flaky run.","rubricLabels":["Uses these 3 groups (names may differ): imports a package or module that isn't available #1, #4, #8, #11 = 4; uses an API or syntax the pinned versions don't support #2, #6 = 2; wrong return value, type or behavior vs the test #3, #7, #9, #10, #12, #13 = 6","Leaves out #5 and ends with a line saying Total: 12","Each category line lists its # ids followed by the count, and each count equals the number of ids on that line"]}