{"id":"data2-sample-ratio-mismatch-readout","audience":"product-managers","category":"data","track":"data","technique":"verification","level":"advanced","title":"Spot a lopsided test split before trusting the lift","free":false,"task":"Your analyst reports a huge lift in a test, but the arms look unbalanced. Write exactly 4 labeled lines in plain text, under 110 words: Overall, By segment, Problem and Decision. Show the conversion math for each segment and arm, explain the mix problem in plain words, and make Decision start with Do not trust or Trust.","materialLabel":"Test results (attached to your prompt automatically)","material":"Experiment: New recommendations carousel (Maple Marketplace), planned 50/50 split\n\nResults by user type:\n\nNew visitors\n- Control: 1,000 visitors, 100 purchases\n- Variant: 200 visitors, 22 purchases\n\nReturning visitors\n- Control: 200 visitors, 60 purchases\n- Variant: 1,000 visitors, 320 purchases\n\nTotals:\n- Control: 1,200 visitors, 160 purchases\n- Variant: 1,200 visitors, 342 purchases\n\nDashboard headline: \"Variant converts 28.5% vs 13.3% - a 15-point lift!\"\n\nNotes:\n- The plan was a random 50/50 split inside each user type. Instead, variant got mostly returning visitors and control mostly new visitors.\n- Returning visitors buy more often than new visitors anyway.\n- The assignment tool had a bug on Nov 10 when a cache was refreshed; engineering has not confirmed whether it is fixed.\n- The VP wants a decision by Friday.","rules":[{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Celebrating the lift","why":"The AI accepts the 15-point lift because the dashboard says so.","fix":"Add: \"Check the arms are comparable; start Decision with Do not trust if not.\"","checks":["r0"]},{"title":"Skipping segment math","why":"The AI says 'segments differ' without the four divisions.","fix":"Add: \"By segment: 100 / 1,000 = 10.0% vs 22 / 200 = 11.0%, and so on.\"","checks":["r1"]},{"title":"Missing the cause","why":"The AI doesn't say the split was lopsided, or blames the carousel.","fix":"Add: \"Problem: the split wasn't 50/50 within each user type.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Plain text, no bold or headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead, for example \"each line under 15 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Gives the four labels","patterns":["overall:","by segment:","problem:","decision:"]},{"label":"Asks for the segment math","patterns":["100 ?/ ?1,?000","division","each arm"]},{"label":"Names the split problem","patterns":["50/50","not comparable","mix"]},{"label":"Fixes the decision wording","patterns":["do not trust"]},{"label":"Caps length","patterns":["under \\d+ words","no notes"]}],"hint":"A suspiciously large lift calls for a look at who was in each arm. Ask for the segment math, not just the total.","mission":"Is this A/B test result trustworthy?","musts":["Segment math for each arm","Explain the mix problem","Do not trust decision"],"rubricLabels":["By segment shows the arithmetic for each arm: new visitors 100 / 1,000 = 10.0% vs 22 / 200 = 11.0%, and returning visitors 60 / 200 = 30.0% vs 320 / 1,000 = 32.0%","Problem says the split was not 50/50 inside each user type (variant got mostly returning visitors, control mostly new), so the 28.5% vs 13.3% gap comes mostly from who was in each arm, not from the carousel","Decision starts with 'Do not trust' the 15-point lift and says to fix and confirm the assignment bug before re-running, and the reply has exactly four lines labeled Overall:, By segment:, Problem: and Decision:"]}