Library › AI Evals › Software developers
File a bug report when an AI PR summary misses a breaking change
Practice: Asking for a format you can use. About 5 minutes.
About this exercise
Your AI PR summarizer said 'No breaking changes' on a PR that removed an API field, and a client integration broke. Write exactly 6 numbered lines: Title, Input (PR and the key diff line), Expected, Actual (quote), Severity, and Likely cause. Use the platform team's scale: P1 = missed a breaking change or security issue, P2 = wrong but harmless, P3 = wording. Base the cause on the logs. Plain text, under 160 words.
Common mistakes when you ask AI for this
- Guessing the severity. The AI uses its own scale and calls it 'High' or 'Major'. Try: Give the scale: "P1 = missed a breaking change or security issue."
- Fuzzy input and actual. The AI writes 'the summary was wrong' without the field or the quote. Try: Say: "Name the removed field and quote the summary."
- Repeating the Slack guess. The AI echoes 'the model isn't smart enough' instead of reading the truncation warning. Try: Add: "Base the cause on the logs, not Slack opinions."
- A long postmortem. The AI writes a timeline, impact and action items. Try: Be exact: "Exactly 6 numbered lines."
What a strong prompt includes
- Lists the fields
- Gives the severity scale
- Points to the logs
- Says exactly how many
- Asks for plain text
Related exercises
- Adapt one ad image to 3 sizes with Midjourney (Image & Video AI · Marketing and content)
- Announce a referral bonus with the steps to claim it (Marketing AI · HR and recruiting)
- Announce a team meeting change people will actually follow (Writing AI · Managers and team leads)
- Answer a 'found an error' question in STAR order (Writing AI · Finance and accounting)
- Answer a wholesale inquiry with clear terms (Writing AI · Small business owners)