Library › AI Evals › Software developers
Group 13 failing code-assistant runs into 3 causes
Practice: Asking for a format you can use. About 5 minutes.
About this exercise
You reviewed 13 failing runs from your code assistant's eval suite and left a note on each. Group them into exactly 3 numbered failure categories, each line 'Category: #ids = count', then one last line 'Total: N'. Leave out any run that passed when re-run (flaky). Plain text, under 70 words.
Common mistakes when you ask AI for this
- Too many categories. The AI makes a group per error type, so nothing points to one fix. Try: Say: "Exactly 3 categories."
- No ids or counts. The AI explains the errors but you can't see which runs belong where. Try: Show the shape: "Category: #1, #4 = 2".
- Counting the flaky run. The AI counts every note, including the one that passed on re-run. Try: Add: "Leave out runs that passed on re-run, and end with Total: N."
- Getting markdown back. AI defaults to bold headers and code formatting. Try: Ask for "plain text, no bold, no headings."
What a strong prompt includes
- Says exactly how many
- Asks for ids and counts
- Handles the flaky run
- Asks for a total
- Asks for plain text
Related exercises
- Announce a referral bonus with the steps to claim it (Marketing AI · HR and recruiting)
- Announce a team meeting change people will actually follow (Writing AI · Managers and team leads)
- Answer a 'found an error' question in STAR order (Writing AI · Finance and accounting)
- Answer a wholesale inquiry with clear terms (Writing AI · Small business owners)
- Answer an RFP timeline question in exactly four phases (Writing AI · Sales and customer success)