{"id":"eval-pm-bug-report-translated-price","audience":"product-managers","category":"evaluation","track":"evals","technique":"format","title":"File a bug report for an AI translation that broke a price","free":false,"task":"Your AI translated a German product page and turned the price into the wrong number. Write exactly 6 numbered lines: Title, Input (the source text), Expected, Actual, Severity and Likely cause. Use the localization team's scale: Sev1 = a price, legal or safety text is wrong, Sev2 = meaning changed, Sev3 = wording only. Base the cause on the logs, not guesses. Plain text, under 160 words.","materialLabel":"The source, the AI translation and the logs (attached to your prompt automatically)","material":"Kettleby Kitchen - product page auto-translation, de-DE to en-US\nPage: SKU KK-4410 \"Gusseisen-Bräter 28 cm\"\n\nSOURCE (de-DE)\n\"Gusseisen-Bräter, 28 cm, 4,2 Liter. Preis: 1.299,00 €. Spülmaschinengeeignet: nein.\"\n\nAI OUTPUT (en-US, shown live Nov 3-5)\n\"Cast iron roaster, 28 cm, 4.2 liters. Price: €1.29. Dishwasher safe: no.\"\n\nLOGS (translate-svc)\nINFO job=tr-88124 model=translate-mini prompt=v7 locale=de-DE->en-US\nWARN number_normalizer: parsed \"1.299,00\" with en-US rules (period = decimal)\nINFO post_check=skipped (price fields not in check list since v7)\n\nSlack\nLena (merch): \"the AI is bad at German, let's use a human agency\"\nDev: \"v7 removed price fields from the post-check to cut latency\"\nSupport: 41 orders at €1.29 before the page was pulled. Finance is reviewing.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":160}],"mistakes":[{"title":"Guessing the severity","why":"The AI picks 'Critical' or 'High' on its own scale, not the localization team's.","fix":"Give the scale: \"Sev1 = a price, legal or safety text is wrong.\"","checks":["r0"]},{"title":"Fuzzy expected vs actual","why":"The AI writes 'price was incorrect' without the two numbers.","fix":"Say: \"Expected and Actual give the exact prices and the job id.\"","checks":["r1"]},{"title":"Repeating the Slack guess","why":"The AI echoes 'the AI is bad at German' instead of reading the log warning.","fix":"Add: \"Base the cause on the logs, not Slack opinions.\"","checks":["r2"]},{"title":"A long incident write-up","why":"The AI adds impact, timeline and next steps until it's a page long.","fix":"Be exact: \"Exactly 6 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Lists the fields","patterns":["expected","actual","severity","title"]},{"label":"Gives the severity scale","patterns":["sev1","price","scale"]},{"label":"Points to the logs","patterns":["log","normaliz","not.{0,20}guess"]},{"label":"Says exactly how many","patterns":["\\b(6|six)\\b.{0,25}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good AI bug report shows the exact wrong output, ranks it on the team's scale, and points at the evidence. Name the fields, give the scale, and point the AI at the logs.","rubricLabels":["Severity is Sev1 because a price was wrong","Expected says the price should be €1,299.00 (or 1,299.00 €) and Actual says the page showed €1.29, naming job tr-88124 or SKU KK-4410","Likely cause points to the number normalizer reading the German number with en-US rules and the v7 post-check skipping price fields, not to the model being bad at German"]}