{"id":"data2-two-metrics-conflict-ship-decision","audience":"product-managers","category":"decisions","track":"data","technique":"verification","level":"advanced","title":"Make a ship decision when two metrics move in opposite directions","free":false,"task":"Your pricing-display test raised revenue per visitor but lowered conversion. Write exactly 4 labeled lines in plain text, under 100 words: Primary, Guardrail, Decision and Risk. Show each difference with its math and apply the pre-registered rule, not the Slack opinions. Decision must start with Ship or Do not ship.","materialLabel":"Test summary and rule (attached to your prompt automatically)","material":"Experiment: Show price per month vs total price at checkout (Maple Marketplace), 21 days, both arms above the planned sample, test tool marks both results reliable.\n\nPre-registered rule (agreed Oct 20):\n- Primary metric: revenue per visitor. Ship only if it rises by at least 2%.\n- Guardrail: conversion rate must not fall by more than 0.5 points.\n- If both pass, ship.\n\nResults:\n- Revenue per visitor: control $3.10, variant $3.22 (+$0.12).\n- Conversion rate: control 5.0%, variant 4.6% (-0.4 points).\n- Refund rate: unchanged.\n\nSlack:\n- Growth lead: \"Conversion is down, I don't like it. Let's not ship.\"\n- Finance: \"Revenue per visitor is what pays the bills.\"\n- PM: \"Let's follow the rule.\"\n\nRisk notes:\n- The conversion drop is within the guardrail but is the largest decline in any test this year.\n- The variant has only been seen by US visitors; international visitors weren't in the test.","rules":[{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Siding with the growth lead","why":"The AI refuses to ship because conversion fell.","fix":"Add: \"Apply the rule: ship if revenue per visitor is up 2%+ and conversion fell 0.5 points or less.\"","checks":["r0"]},{"title":"Percent math skipped","why":"The AI says revenue per visitor is up without computing 3.9%.","fix":"Add: \"Primary: +$0.12 / $3.10 = +3.9%.\"","checks":["r1"]},{"title":"No risk line","why":"The AI declares victory.","fix":"Add: \"Risk: the conversion drop is the biggest this year, or international isn't tested.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Plain text, no bold or headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead, for example \"each line under 15 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Names the four labels","patterns":["primary:","guardrail:","decision:","risk:"]},{"label":"Applies the rule","patterns":["pre.?registered","rule"]},{"label":"Shows the math","patterns":["0\\.12 ?/ ?\\$?3\\.10","3\\.9%","-0\\.4"]},{"label":"Fixes the decision wording","patterns":["start with ship","ship or do not ship"]},{"label":"Ignores Slack","patterns":["ignore the slack","opinions"]}],"hint":"A pre-registered rule with two conditions needs each tested with its own math before the decision.","mission":"Should we ship this change?","musts":["Both conditions tested","Follow the written rule","Name a risk"],"rubricLabels":["Primary shows $3.10 to $3.22 is +$0.12, which is about +3.9% ($0.12 / $3.10), above the 2% bar","Guardrail shows conversion 5.0% to 4.6% is -0.4 points, within the 0.5-point limit, and Decision starts with 'Ship' because both conditions pass under the pre-registered rule","Risk mentions that the conversion drop is the largest of the year or that international visitors were not tested, and the reply has exactly four lines labeled Primary:, Guardrail:, Decision: and Risk:"]}