Library › AI Evals › Software developers

File a bug report when an AI PR summary misses a breaking change

Practice: Asking for a format you can use. About 5 minutes.

About this exercise

Your AI PR summarizer said 'No breaking changes' on a PR that removed an API field, and a client integration broke. Write exactly 6 numbered lines: Title, Input (PR and the key diff line), Expected, Actual (quote), Severity, and Likely cause. Use the platform team's scale: P1 = missed a breaking change or security issue, P2 = wrong but harmless, P3 = wording. Base the cause on the logs. Plain text, under 160 words.

Common mistakes when you ask AI for this

  • Guessing the severity. The AI uses its own scale and calls it 'High' or 'Major'. Try: Give the scale: "P1 = missed a breaking change or security issue."
  • Fuzzy input and actual. The AI writes 'the summary was wrong' without the field or the quote. Try: Say: "Name the removed field and quote the summary."
  • Repeating the Slack guess. The AI echoes 'the model isn't smart enough' instead of reading the truncation warning. Try: Add: "Base the cause on the logs, not Slack opinions."
  • A long postmortem. The AI writes a timeline, impact and action items. Try: Be exact: "Exactly 6 numbered lines."

What a strong prompt includes

  • Lists the fields
  • Gives the severity scale
  • Points to the logs
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises