Library › Data AI › Product managers

Decide whether a fading test lift is worth shipping

  • For Product managers
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Advanced

About this exercise

A test's lift shrank in week 2 and your VP wants a decision. Write exactly 4 labeled lines in plain text, under 100 words: Week 1, Week 2, Rule and Decision. Show each week's relative lift as (variant - control) / control, apply the written rule to the final week only, and make Decision start with Ship, Extend or Stop.

Common mistakes when you ask AI for this

  • Shipping on week 1. The AI averages the weeks or ships on the +10% headline. Try: Add: "Judge the final week only."
  • Absolute not relative. The AI reports 0.4 and 0.1 points but not the relative percents. Try: Add: "Relative lift = (variant - control) / control."
  • Wrong verdict. The AI says Stop or Ship instead of Extend. Try: Add: "Above 0% but under 5% means Extend one more week."
  • Leaving the format to chance. The AI picks its own length and layout unless told the exact shape. Try: Add: "Plain text, no bold or headings."
  • Trusting the AI to count words. AI is bad at counting. Ask for a total word limit and it often lands over it. Try: Cap each part instead, for example "each line under 15 words."

What a strong prompt includes

  • Gives the four labels
  • Asks for relative lift
  • Says final week only
  • Gives the cut-offs
  • Ignores Slack

Practice this now ↑

Related exercises