Library › Data AI › Product managers

Write an A/B test readout as a JSON block

  • For Product managers
  • You practice Asking for a format you can use
  • Time about 5 minutes
  • Level Advanced

About this exercise

Your experimentation tool ingests readouts as JSON. Summarize the checkout test as one JSON block (a single code block) with exactly these keys: decision, control_rate, variant_rate, lift_points, users_per_arm, reason. Rates as numbers like 12.0 for percent, decision one of ship, no_ship or extend according to the pre-registered plan, and reason one sentence under 25 words. At most 15 words of text outside the code.

Common mistakes when you ask AI for this

  • Following the hype. The AI goes with 'promising, ship it' instead of the written plan. Try: Add: "Apply the pre-registered plan: under 3,000 users per arm means extend."
  • Wrong or extra keys. The AI adds fields like p_value and notes or renames keys. Try: Add: "Exactly these keys: decision, control_rate, variant_rate, lift_points, users_per_arm, reason."
  • Fuzzy reason. The AI writes a long reason that mixes opinions. Try: Add: "reason: one sentence saying the 3,000-per-arm minimum was not reached."
  • Leaving the format to chance. The AI picks its own length and layout unless told the exact shape. Try: Add: "Put the code in exactly 1 code block. Keep the writing outside the code under 15 words."

What a strong prompt includes

  • Lists the keys
  • Applies the plan
  • Gives the rate math
  • Asks for valid JSON in one block
  • Limits outside text

Practice this now ↑

Related exercises