{"id":"pm-experiment-readout-decision","audience":"product-managers","category":"data","track":"data","technique":"grounding","title":"Write an A/B test readout that follows the decision rule","free":false,"task":"You're posting the onboarding checklist test result for your CEO, who is not a stats person and wants a clear call she can repeat. You want it under 90 words, plain text, and the first sentence must state the ship decision according to the rule the team agreed before the test. Give the primary metric's two rates and say in plain words whether the difference is reliable, with no stats jargon, and leave the team debate (who wanted what) out.","materialLabel":"Test results and thread (attached to your prompt automatically)","material":"Experiment: Onboarding checklist (Tallyroom), Oct 6 - Oct 27\n8,400 new signups, split 50/50.\n\nPre-registered plan (agreed Oct 1):\n- Primary metric: % of new accounts that invite a teammate within 7 days.\n- Decision rule: ship only if the primary metric improves AND p < 0.05.\n- Guardrail: support tickets per new account must not rise more than 10%.\n\nResults (analyst: Wen):\n- Primary: control 22.4%, checklist 23.1% (+0.7 points), p = 0.31.\n- Secondary: time in app in first week +18% (p = 0.01).\n- Secondary: 61% of checklist users completed all steps.\n- Guardrail: tickets per new account +4% (within limit).\n- Note: 2 days in week 1 had a tracking bug; those days are already excluded above.\n\nThread:\nLou (design lead): Time in app is up 18%!! This is a huge win, let's ship it.\nWen: Reminder that time in app wasn't the decision metric.\nRina (CEO): I just want a clear call I can repeat.\nLou: Can we at least ship to 50%?","rules":[{"type":"maxWords","max":90},{"type":"plainText"}],"mistakes":[{"title":"Asking to 'write up the results'","why":"The AI writes a long report with headings for each metric and a discussion section.","fix":"Limit it: \"4 sentences, plain text, no headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead: \"4 sentences, each under 22 words.\"","checks":["maxWords"]},{"title":"Burying the call","why":"The AI walks through the numbers first, so the CEO has to hunt for the decision.","fix":"Add: \"First sentence: the ship decision under our pre-registered rule.\"","checks":["r0"]},{"title":"Using stats jargon","why":"The AI reports p-values and significance, which the CEO can't repeat to anyone.","fix":"Add: \"No stats terms. Say in plain words whether the difference is reliable.\"","checks":["r1"]},{"title":"Retelling the debate","why":"The AI covers who wanted to ship and why, so the readout sounds undecided.","fix":"Add: \"Time in app is secondary. Leave out the ship-it debate and the 50% idea.\"","checks":["r2"]}],"promptHints":[{"label":"Puts the decision first","patterns":["first sentence","lead with","start with","decision first"]},{"label":"Names the decision rule","patterns":["rule","pre.?register","p ?< ?0?\\.05","primary metric"]},{"label":"Bans stats jargon","patterns":["jargon","plain (words|english|language)","no p.?value","not a stats","non.?technical"]},{"label":"Limits the length","patterns":["\\d+\\s*words","\\b(4|four)\\b.{0,10}sentence","word (limit|count)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A readout is a decision, not a tour of the data. Tell the AI the agreed rule, that the call goes first, which numbers to report, and which metric is only secondary.","rubricLabels":["The first sentence says the checklist does not ship (or is not shipped as tested) because the primary metric did not meet the agreed rule","Reports the primary metric as 22.4% vs 23.1% and says in plain words that the difference is too small to be sure it is real, without stats terms (no p-value, p =, statistically significant, confidence)","If time in app appears, it is called a secondary metric that did not drive the decision, and the output does not mention the design lead's push to ship or the 50% rollout idea"]}