Library › Data AI › Product managers
Report an A/B test result without the buggy first days
Practice: Giving the right context. About 5 minutes.
About this exercise
You're posting the checkout A/B test result. The variant had a bug on days 1-2, so leave both groups' day 1-2 numbers out. Write exactly 3 numbered lines: 1) control conversion rate, 2) variant conversion rate, each as conversions ÷ visitors = X%, 3) the difference in percentage points. Don't call it statistically significant. Plain text, under 50 words.
Common mistakes when you ask AI for this
- Not mentioning the bug days. The AI uses all 14 days, so the variant looks weaker than it is. Try: Add: "Leave out days 1-2 for both groups."
- Claiming significance. The AI repeats Jordan's claim though no test was run. Try: Add: "Don't call it significant."
- Not asking to see the work. Bare rates hide which days were counted. Try: Add a line to copy: "1. Control: 320 ÷ 8,000 = 4.0%"
- Trusting the AI to count words. AI is bad at counting. Ask for a total word limit and it often lands over it. Try: Cap each line: "each line under 12 words."
- Forgetting where it goes. AI loves markdown tables and bold, which show up as clutter in a product channel. Try: Add: "Plain text, no bold, headings or tables."
What a strong prompt includes
- Removes the bug days
- Asks for points
- Avoids significance claims
- Asks to show the math
- Asks for plain text
Related exercises
- Announce a hybrid work policy change with clear actions (Writing AI · HR and recruiting)
- Announce a new weekend rotation in three clear rules (Productivity AI · Managers and team leads)
- Announce a promotion with the right date and no pay details (Writing AI · HR and recruiting)
- Announce a team change and answer the first question (Writing AI · Managers and team leads)
- Answer a showing request for a home that's under contract (Writing AI · Real estate agents)