Library › Data AI › Product managers
Decide whether a fading test lift is worth shipping
- For Product managers
- You practice Catching mistakes and made-up facts
- Time about 5 minutes
- Level Advanced
About this exercise
A test's lift shrank in week 2 and your VP wants a decision. Write exactly 4 labeled lines in plain text, under 100 words: Week 1, Week 2, Rule and Decision. Show each week's relative lift as (variant - control) / control, apply the written rule to the final week only, and make Decision start with Ship, Extend or Stop.
Common mistakes when you ask AI for this
- Shipping on week 1. The AI averages the weeks or ships on the +10% headline. Try: Add: "Judge the final week only."
- Absolute not relative. The AI reports 0.4 and 0.1 points but not the relative percents. Try: Add: "Relative lift = (variant - control) / control."
- Wrong verdict. The AI says Stop or Ship instead of Extend. Try: Add: "Above 0% but under 5% means Extend one more week."
- Leaving the format to chance. The AI picks its own length and layout unless told the exact shape. Try: Add: "Plain text, no bold or headings."
- Trusting the AI to count words. AI is bad at counting. Ask for a total word limit and it often lands over it. Try: Cap each part instead, for example "each line under 15 words."
What a strong prompt includes
- Gives the four labels
- Asks for relative lift
- Says final week only
- Gives the cut-offs
- Ignores Slack
Related exercises
- Accrue unpaid wages for the working days after the last pay period (Finance AI · Accountants)
- Age a returns queue in business days and flag the ones over the limit (Customer Support & Success AI · Customer support teams)
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Annualize recurring revenue without the one-time fee (Data AI · Finance and accounting)
- Answer a credit dispute with the full-hour rule and the math (Customer Support & Success AI · Customer support teams)