{"id":"data2-novelty-effect-fading-lift-decision","audience":"product-managers","category":"decisions","track":"data","technique":"verification","level":"advanced","title":"Decide whether a fading test lift is worth shipping","free":false,"task":"A test's lift shrank in week 2 and your VP wants a decision. Write exactly 4 labeled lines in plain text, under 100 words: Week 1, Week 2, Rule and Decision. Show each week's relative lift as (variant - control) / control, apply the written rule to the final week only, and make Decision start with Ship, Extend or Stop.","materialLabel":"Weekly test results and rule (attached to your prompt automatically)","material":"Test: New homepage hero (Maple Marketplace), 2 weeks, 50/50 split\nMetric: signup rate on the homepage.\n\nWeek 1: control 4.0%, variant 4.4%\nWeek 2: control 4.0%, variant 4.1%\n\nBoth weeks have similar traffic (about 20,000 visitors per arm per week).\n\nPre-registered rule (agreed before launch):\n- Judge the test on the final week only, because early weeks can be inflated by novelty.\n- Ship if the final week's relative lift is 5% or more.\n- Extend one more week if the final week's relative lift is above 0% but under 5%.\n- Stop if the final week's relative lift is 0% or below.\n\nSlack:\n- Designer: \"Week 1 was +10%! Let's ship.\"\n- Analyst: \"Week 2 is what matters under the rule.\"\n\nThe VP wants a one-line decision.","rules":[{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Shipping on week 1","why":"The AI averages the weeks or ships on the +10% headline.","fix":"Add: \"Judge the final week only.\"","checks":["r0"]},{"title":"Absolute not relative","why":"The AI reports 0.4 and 0.1 points but not the relative percents.","fix":"Add: \"Relative lift = (variant - control) / control.\"","checks":["r1"]},{"title":"Wrong verdict","why":"The AI says Stop or Ship instead of Extend.","fix":"Add: \"Above 0% but under 5% means Extend one more week.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Plain text, no bold or headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead, for example \"each line under 15 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Gives the four labels","patterns":["week 1:","week 2:","rule:","decision:"]},{"label":"Asks for relative lift","patterns":["relative","\\(variant ?- ?control\\)"]},{"label":"Says final week only","patterns":["final week"]},{"label":"Gives the cut-offs","patterns":["5%","0%"]},{"label":"Ignores Slack","patterns":["ignore the slack","opinions"]}],"hint":"A fading lift tempts a ship call from the best week. Name the rule that judges the final week.","mission":"Should we ship the new homepage hero?","musts":["Relative lift math each week","Final week only","Extend decision"],"rubricLabels":["Week 1 shows (4.4 - 4.0) / 4.0 = +10.0% and Week 2 shows (4.1 - 4.0) / 4.0 = +2.5%","Rule states that only the final week counts: 5% or more ships, above 0% but under 5% extends one more week, 0% or less stops","Decision starts with 'Extend' (one more week) because the final week's +2.5% is under 5%, not with Ship, and the reply has exactly four lines labeled Week 1:, Week 2:, Rule: and Decision:"]}