Library › Data AI › Product managers

Write an A/B test readout that follows the decision rule

Practice: Keeping the AI to your sources. About 5 minutes.

About this exercise

You're posting the onboarding checklist test result for your CEO, who is not a stats person and wants a clear call she can repeat. You want it under 90 words, plain text, and the first sentence must state the ship decision according to the rule the team agreed before the test. Give the primary metric's two rates and say in plain words whether the difference is reliable, with no stats jargon, and leave the team debate (who wanted what) out.

Common mistakes when you ask AI for this

  • Asking to 'write up the results'. The AI writes a long report with headings for each metric and a discussion section. Try: Limit it: "4 sentences, plain text, no headings."
  • Trusting the AI to count words. AI is bad at counting. Ask for a total word limit and it often lands over it. Try: Cap each part instead: "4 sentences, each under 22 words."
  • Burying the call. The AI walks through the numbers first, so the CEO has to hunt for the decision. Try: Add: "First sentence: the ship decision under our pre-registered rule."
  • Using stats jargon. The AI reports p-values and significance, which the CEO can't repeat to anyone. Try: Add: "No stats terms. Say in plain words whether the difference is reliable."
  • Retelling the debate. The AI covers who wanted to ship and why, so the readout sounds undecided. Try: Add: "Time in app is secondary. Leave out the ship-it debate and the 50% idea."

What a strong prompt includes

  • Puts the decision first
  • Names the decision rule
  • Bans stats jargon
  • Limits the length
  • Asks for plain text

Practice this now ↑

Related exercises