{"id":"eval-dev-bug-report-dropped-billing-issue","audience":"developers","category":"evaluation","track":"evals","technique":"format","title":"Write a bug report for a summary that dropped a charge","free":false,"task":"Your ticket summarizer dropped a customer's double-charge complaint, and you need a bug report for the tracker. Write exactly 6 numbered lines labeled Title, Input, Expected, Actual, Severity and Likely cause. Use the team scale: P1 means a money or safety issue was missed, P2 a wrong but harmless summary, P3 style only. Base the cause on the logs, not guesses. Plain text, under 200 words.","materialLabel":"The ticket, the summary and the logs (attached to your prompt automatically)","material":"TICKET #T-4471 (customer: Hollis Bakery, plan: Pro)\n\"Hi, three things. First, the invoice PDF won't download in Safari, it just spins. Second, can you add my bookkeeper (ana@hollisbakery.example) as a viewer? Third, and most important, I was charged $49 twice on Nov 2 for the same Pro renewal. Please refund one.\" [the full ticket also pasted 1,900 words of email history above this]\n\nAI SUMMARY SHOWN TO AGENT (prompt v3.2)\n\"Customer can't download invoice PDF in Safari and wants to add a bookkeeper as a viewer. Priority: normal.\"\n\nLOGS (summarizer-svc, Nov 3 14:02)\nINFO  model=summarizer-small prompt=v3.2 ticket=T-4471\nWARN  input truncated to 2000 tokens (original 2614 tokens), kept first 2000\nINFO  summary_tokens=31 latency_ms=812\n\nSlack\nPriya: probably the model is just dumb, let's switch to the big one\nDev lead: we changed truncation from \"keep last\" to \"keep first\" in v3.2 last Tuesday\nAgent Rosa: the customer got angry after waiting 2 days for the refund","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":200}],"mistakes":[{"title":"Guessing the severity","why":"The AI picks 'High' or 'Medium' on its own scale because it doesn't know your team's P1-P3 rules.","fix":"Give the scale: \"P1 = money or safety missed.\"","checks":["r0"]},{"title":"Fuzzy expected vs actual","why":"The AI writes 'summary was incomplete' without saying what was missing.","fix":"Say: \"Expected and Actual name the exact missing detail and the ticket id.\"","checks":["r1"]},{"title":"Repeating the Slack guess","why":"The AI echoes 'switch to a bigger model' instead of reading the truncation warning.","fix":"Add: \"Base the cause on the logs, not opinions in Slack.\"","checks":["r2"]},{"title":"A long write-up","why":"The AI adds steps to reproduce, environment and next actions until it's a page long.","fix":"Say: \"Exactly 6 numbered lines: Title, Input, Expected, Actual, Severity, Likely cause.\"","checks":["numberedItems","maxWords"]},{"title":"Getting markdown back","why":"AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker.","fix":"Ask for \"plain text, no bold, no headings, no tables.\"","checks":["plainText"]}],"promptHints":[{"label":"Lists the fields","patterns":["expected","actual","severity","title"]},{"label":"Gives the severity scale","patterns":["P1","money","scale"]},{"label":"Points to the logs","patterns":["log","truncat","not.{0,20}guess"]},{"label":"Says how many lines","patterns":["\\b(6|six)\\b.{0,20}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good AI bug report lets someone reproduce the failure and rank it without reading the whole ticket. Name the fields, give your severity scale, and point the AI at the logs.","rubricLabels":["The Severity line says P1 because the summary missed a money issue (the $49 double charge)","Expected says the summary should include the $49 double charge and refund request, and Actual says the summary left it out, naming ticket T-4471 or prompt v3.2","Likely cause points to the input being cut to the first 2000 tokens (the v3.2 change to keep the first part), not to the model being too small"]}