{"id":"dat-pm-ab-test-bug-days","audience":"product-managers","category":"data","technique":"context","title":"Report an A/B test result without the buggy first days","free":false,"task":"You're posting the checkout A/B test result. The variant had a bug on days 1-2, so leave both groups' day 1-2 numbers out. Write exactly 3 numbered lines: 1) control conversion rate, 2) variant conversion rate, each as conversions ÷ visitors = X%, 3) the difference in percentage points. Don't call it statistically significant. Plain text, under 50 words.","materialLabel":"Experiment export (attached to your prompt automatically)","material":"EXPERIMENT: one-page checkout (Oct 20 - Nov 2)\n\nTotals (all 14 days)\nControl: 10,000 visitors, 400 conversions\nVariant: 10,000 visitors, 480 conversions\n\nDays 1-2 only (Oct 20-21)\nControl: 2,000 visitors, 80 conversions\nVariant: 2,000 visitors, 40 conversions\n\nNotes:\n- Oct 21 evening: fixed a bug where the variant's promo code field didn't submit.\n- Dashboard headline says \"Variant +20% lift!\" (uses all 14 days).\n- Jordan (PM) posted \"it's significant!!\" in the channel. No significance test has been run.\n- Mobile was 60% of traffic in both groups.\n- The experiment split traffic 50/50 between control and variant.\n- Conversion means a completed purchase.\n- Data science will run the significance test next week.\n- Engineering wants to know whether to keep the one-page checkout.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":50}],"mistakes":[{"title":"Not mentioning the bug days","why":"The AI uses all 14 days, so the variant looks weaker than it is.","fix":"Add: \"Leave out days 1-2 for both groups.\"","checks":["r0","r1"]},{"title":"Claiming significance","why":"The AI repeats Jordan's claim though no test was run.","fix":"Add: \"Don't call it significant.\"","checks":["r2"]},{"title":"Not asking to see the work","why":"Bare rates hide which days were counted.","fix":"Add a line to copy: \"1. Control: 320 ÷ 8,000 = 4.0%\"","checks":["numberedItems","r0"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each line: \"each line under 12 words.\"","checks":["maxWords"]},{"title":"Forgetting where it goes","why":"AI loves markdown tables and bold, which show up as clutter in a product channel.","fix":"Add: \"Plain text, no bold, headings or tables.\"","checks":["plainText"]}],"promptHints":[{"label":"Removes the bug days","patterns":["day(s)? 1-2|first (2|two) days","bug","remove|exclude|leave out"]},{"label":"Asks for points","patterns":["points?","difference"]},{"label":"Avoids significance claims","patterns":["significan"]},{"label":"Asks to show the math","patterns":["show (the |your )?(math|work)","x \\$|\\bx\\b.{0,10}=","calculation","step"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no (headings|tables)"]}],"hint":"The totals include days when the variant was broken. Tell the AI to remove them, and how to state the difference.","rubricLabels":["Control is 320 ÷ 8,000 = 4.0% (days 1-2 removed)","Variant is 440 ÷ 8,000 = 5.5% (days 1-2 removed)","The difference is +1.5 percentage points, and the output does not call the result statistically significant or repeat the +20% headline"]}