{"id":"data2-segment-ab-results-email-design-lead","audience":"product-managers","category":"data","track":"data","technique":"tone","level":"intermediate","title":"Email segment-level test results without overclaiming","free":false,"task":"Your design lead, Kofi Mensah, wants to know how the new onboarding screen did. Write a plain-text email under 100 words with a Subject: line and three short labeled parts: What we saw:, What we can't say yet: and Next:. Report each segment's two rates, call a result real only where the notes say it is reliable, and don't promise the result will hold.","materialLabel":"Test results (attached to your prompt automatically)","material":"Onboarding screen test - Orbit Notes, Nov 3-17\n\nMobile signups\n- Old screen: 4.0% finish setup (n = 6,000 per arm)\n- New screen: 5.2% finish setup\n- The test tool marks this difference reliable.\n\nDesktop signups\n- Old screen: 6.0% finish setup (n = 2,000 per arm)\n- New screen: 6.1% finish setup\n- The test tool says this difference is too small to tell apart from chance.\n\nOverall (both devices): 4.5% old vs 5.4% new.\n\nOther notes:\n- Mobile is 75% of signups.\n- The test ran 14 days, covering two weekends.\n- The plan for next week: roll the new screen out to mobile only, and run a longer desktop test.\n- Kofi is a designer: plain words, no stats terms.","rules":[{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Treating every lift as real","why":"The AI says both devices improved.","fix":"Add: \"Mobile is a real gain; desktop 6.0% to 6.1% is no real difference.\"","checks":["r0"]},{"title":"Using stats terms","why":"The AI mentions p-values and confidence levels to a designer.","fix":"Add: \"No stats terms.\"","checks":["r1"]},{"title":"Overpromising","why":"The AI says the new screen will lift signups by 20%.","fix":"Add: \"We can't promise the mobile gain will hold.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Plain text, no bold or headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead, for example \"each line under 15 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Names the reader","patterns":["kofi","design"]},{"label":"Gives the three labels","patterns":["what we saw:","what we can.t say yet:","next:"]},{"label":"Gives the segment rates","patterns":["4\\.0%","5\\.2%","6\\.1%"]},{"label":"Rules out jargon","patterns":["no stats","plain words"]},{"label":"Gives the plan","patterns":["mobile only","longer desktop"]}],"hint":"Segment results need honesty about which differences are real. Give the AI which are and the plain-words rule.","mission":"Email the design lead the test results.","musts":["Three labeled parts","Reliable vs unclear","No promises"],"rubricLabels":["What we saw: reports mobile 4.0% to 5.2% as a reliable gain and desktop 6.0% to 6.1% as essentially no difference","What we can't say yet: says desktop is unclear (the 0.1 point difference can't be told apart from chance) and does not promise the mobile gain will hold forever, without stats terms like p-value or significance","Next: says roll the new screen out to mobile only and run a longer desktop test, and the email starts with a Subject: line"]}