Spot a lopsided test split before trusting the lift

About this exercise

Your analyst reports a huge lift in a test, but the arms look unbalanced. Write exactly 4 labeled lines in plain text, under 110 words: Overall, By segment, Problem and Decision. Show the conversion math for each segment and arm, explain the mix problem in plain words, and make Decision start with Do not trust or Trust.

Common mistakes when you ask AI for this

• Celebrating the lift. The AI accepts the 15-point lift because the dashboard says so. Try: Add: "Check the arms are comparable; start Decision with Do not trust if not."
• Skipping segment math. The AI says 'segments differ' without the four divisions. Try: Add: "By segment: 100 / 1,000 = 10.0% vs 22 / 200 = 11.0%, and so on."
• Missing the cause. The AI doesn't say the split was lopsided, or blames the carousel. Try: Add: "Problem: the split wasn't 50/50 within each user type."
• Leaving the format to chance. The AI picks its own length and layout unless told the exact shape. Try: Add: "Plain text, no bold or headings."
• Trusting the AI to count words. AI is bad at counting. Ask for a total word limit and it often lands over it. Try: Cap each part instead, for example "each line under 15 words."

What a strong prompt includes

• Gives the four labels
• Asks for the segment math
• Names the split problem
• Fixes the decision wording
• Caps length

Practice this now ↑

Related exercises

• Accrue unpaid wages for the working days after the last pay period (Finance AI · Accountants)
• Age a returns queue in business days and flag the ones over the limit (Customer Support & Success AI · Customer support teams)
• Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
• Annualize recurring revenue without the one-time fee (Data AI · Finance and accounting)
• Answer a credit dispute with the full-hour rule and the math (Customer Support & Success AI · Customer support teams)