{"id":"data2-ab-guardrail-breach-decision","audience":"product-managers","category":"decisions","track":"data","technique":"verification","level":"intermediate","title":"Make a ship decision when the guardrail failed","free":false,"task":"Your VP needs a go or no-go on the new search layout test. Write exactly 4 labeled lines in plain text, under 90 words: Result, Guardrail, Decision and Next step. Decision must start with Ship or Do not ship and follow the pre-registered rule. Use only the numbers in the notes and show the differences.","materialLabel":"Test summary (attached to your prompt automatically)","material":"Experiment: New search results layout (Tallyroom marketplace), Nov 3 - Nov 17\nSample: 14,000 users per arm (planned: 12,000), so the minimum sample is met.\n\nPre-registered rule (agreed Oct 28):\n- Ship only if the primary metric improves AND no guardrail is breached.\n- Primary metric: search-to-purchase rate.\n- Guardrail: p75 page load time must not rise by more than 0.3 seconds.\n\nResults:\n- Search-to-purchase: control 8.0%, new layout 8.6% (+0.6 points). The test tool marks this reliable.\n- p75 page load: control 2.1 seconds, new layout 2.6 seconds (+0.5 seconds).\n- Revenue per user: +2%, not the decision metric.\n- Customer complaints: no change.\n\nSlack:\n- Design lead: \"Purchases are up, the load time can be fixed after launch.\"\n- Eng lead: \"The extra images cause the slowdown; a fix is about 1 sprint.\"\n\nThe VP wants one clear line to repeat to the exec team.","rules":[{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Following the design lead","why":"The AI ships because purchases went up.","fix":"Add: \"Apply the pre-registered rule: ship only if no guardrail is breached.\"","checks":["r0"]},{"title":"Skipping the numbers","why":"The AI says 'load time is slower' without 2.1 to 2.6 seconds and the 0.3 limit.","fix":"Add: \"Guardrail line: 2.1 to 2.6 seconds, +0.5, limit 0.3.\"","checks":["r1"]},{"title":"Vague next step","why":"The AI suggests more research with no specific action.","fix":"Add: \"Next step: fix the image slowdown (about one sprint) and re-test.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Plain text, no bold or headings.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands over it.","fix":"Cap each part instead, for example \"each line under 15 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Gives the four labels","patterns":["result:","guardrail:","decision:","next step:"]},{"label":"Applies the rule","patterns":["pre.?registered","rule","guardrail"]},{"label":"Shows differences","patterns":["0\\.6","0\\.5","difference"]},{"label":"Starts the decision","patterns":["do not ship","ship or do not ship"]},{"label":"Caps length","patterns":["under \\d+ words","no other text"]}],"hint":"A rule agreed in advance beats the loudest opinion. Tell the AI to apply the written rule and show the differences.","mission":"Should we ship this test?","musts":["Four labeled lines","Apply the written rule","Show the differences"],"rubricLabels":["Result states search-to-purchase rose from 8.0% to 8.6% (+0.6 points), and Guardrail states p75 load time rose from 2.1 to 2.6 seconds (+0.5 seconds), above the 0.3-second limit","Decision starts with 'Do not ship' and says it is because a guardrail was breached under the pre-registered rule, not because the primary metric failed","Next step proposes fixing the load time (the image slowdown, about one sprint) and re-testing, and the reply is exactly four lines labeled Result:, Guardrail:, Decision: and Next step:"]}