{"id":"eval-dev-bug-report-pr-summary-missed-breaking-change","audience":"developers","category":"evaluation","track":"evals","technique":"format","title":"File a bug report when an AI PR summary misses a breaking change","free":false,"task":"Your AI PR summarizer said 'No breaking changes' on a PR that removed an API field, and a client integration broke. Write exactly 6 numbered lines: Title, Input (PR and the key diff line), Expected, Actual (quote), Severity, and Likely cause. Use the platform team's scale: P1 = missed a breaking change or security issue, P2 = wrong but harmless, P3 = wording. Base the cause on the logs. Plain text, under 160 words.","materialLabel":"The PR, the AI summary and the logs (attached to your prompt automatically)","material":"PR #2291 \"Clean up order serializer\" (merged Nov 3)\n\nDIFF (excerpt)\n  api/serializers/order.py\n- \"discount_code\": order.discount_code,\n  \"total_cents\": order.total_cents,\n  tests/test_order.py: 14 lines changed (fixture cleanup)\n  docs/: 3 files reformatted (whitespace)\n\nAI PR SUMMARY (posted as a comment)\n\"Refactors the order serializer and tidies tests and docs. No breaking changes.\"\n\nLOGS (pr-summarizer)\nINFO pr=2291 files=6 diff_lines=1,840\nWARN diff exceeded 1,500 lines; summarized first 1,500 lines only (sorted: docs/, tests/, api/)\nINFO model=sum-small prompt=v2.4\n\nSlack\nAna: \"the model just isn't smart enough for code\"\nDev lead: \"v2.4 started sorting files alphabetically before truncating\"\nSupport: Halvorsen Retail's checkout integration broke on Nov 4; they read discount_code.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":160}],"mistakes":[{"title":"Guessing the severity","why":"The AI uses its own scale and calls it 'High' or 'Major'.","fix":"Give the scale: \"P1 = missed a breaking change or security issue.\"","checks":["r0"]},{"title":"Fuzzy input and actual","why":"The AI writes 'the summary was wrong' without the field or the quote.","fix":"Say: \"Name the removed field and quote the summary.\"","checks":["r1"]},{"title":"Repeating the Slack guess","why":"The AI echoes 'the model isn't smart enough' instead of reading the truncation warning.","fix":"Add: \"Base the cause on the logs, not Slack opinions.\"","checks":["r2"]},{"title":"A long postmortem","why":"The AI writes a timeline, impact and action items.","fix":"Be exact: \"Exactly 6 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Lists the fields","patterns":["expected","actual","severity","title"]},{"label":"Gives the severity scale","patterns":["P1","breaking","scale"]},{"label":"Points to the logs","patterns":["log","truncat|1,?500","not.{0,20}guess"]},{"label":"Says exactly how many","patterns":["\\b(6|six)\\b.{0,25}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good AI bug report shows the exact miss, ranks it on the team's scale, and points at the evidence. Name the fields, give the scale, and send the AI to the logs.","rubricLabels":["Severity is P1 because the summary missed a breaking change","Input names PR #2291 and the removed discount_code field, and Actual quotes \"No breaking changes\"","Likely cause points to the diff being cut at 1,500 lines after sorting docs/ and tests/ first, so the api/ change was never read, not to the model being too weak"]}