{"id":"data2-ab-readout-json-extend","audience":"product-managers","category":"data","track":"data","technique":"format","level":"advanced","title":"Write an A/B test readout as a JSON block","free":false,"task":"Your experimentation tool ingests readouts as JSON. Summarize the checkout test as one JSON block (a single code block) with exactly these keys: decision, control_rate, variant_rate, lift_points, users_per_arm, reason. Rates as numbers like 12.0 for percent, decision one of ship, no_ship or extend according to the pre-registered plan, and reason one sentence under 25 words. At most 15 words of text outside the code.","materialLabel":"Test summary (attached to your prompt automatically)","material":"Experiment: One-page checkout vs three-step checkout (Brightpath App)\nRun dates: Nov 3 - Nov 10 (7 days)\n\nPre-registered plan (agreed Oct 28):\n- Primary metric: checkout conversion (orders / users who reached checkout).\n- Minimum sample: 3,000 users per arm before any decision.\n- If minimum sample is not reached: decision = extend, regardless of the result.\n- If reached: ship only if the variant is better and the team's test tool says the result is reliable.\n\nResults so far:\n- Control: 2,000 users, 240 orders.\n- Variant: 2,000 users, 276 orders.\n- The test tool says \"result looks promising, 88% confidence\" (the plan wants 95%).\n- Guardrail (refund requests per order) unchanged.\n- Traffic is split evenly; no tracking issues reported.\n\nSlack: \"Looks great, can we just ship?\" - Dev lead\nSlack: \"Let's follow the plan.\" - PM\n\nOutput keys wanted by the tool, all lowercase with underscores.","rules":[{"type":"codeBlocks","count":1},{"type":"maxProseWords","max":15}],"mistakes":[{"title":"Following the hype","why":"The AI goes with 'promising, ship it' instead of the written plan.","fix":"Add: \"Apply the pre-registered plan: under 3,000 users per arm means extend.\"","checks":["r0"]},{"title":"Wrong or extra keys","why":"The AI adds fields like p_value and notes or renames keys.","fix":"Add: \"Exactly these keys: decision, control_rate, variant_rate, lift_points, users_per_arm, reason.\"","checks":["r1"]},{"title":"Fuzzy reason","why":"The AI writes a long reason that mixes opinions.","fix":"Add: \"reason: one sentence saying the 3,000-per-arm minimum was not reached.\"","checks":["r2"]},{"title":"Leaving the format to chance","why":"The AI picks its own length and layout unless told the exact shape.","fix":"Add: \"Put the code in exactly 1 code block. Keep the writing outside the code under 15 words.\"","checks":["codeBlocks","maxProseWords"]}],"promptHints":[{"label":"Lists the keys","patterns":["control_rate","variant_rate","lift_points"]},{"label":"Applies the plan","patterns":["pre.?registered","3,?000","plan"]},{"label":"Gives the rate math","patterns":["240 ?/ ?2,?000","orders / users","x 100"]},{"label":"Asks for valid JSON in one block","patterns":["json","code block"]},{"label":"Limits outside text","patterns":["at most","one (short )?sentence","\\d+ words"]}],"hint":"Structured output needs exact keys and the decision rule from the plan, not from the thread's opinions.","mission":"Report the A/B test result and whether to extend it.","musts":["Exact six keys","Follow the plan, extend","Rates as numbers"],"rubricLabels":["The JSON has exactly the six keys decision, control_rate, variant_rate, lift_points, users_per_arm and reason (no other keys)","decision is \"extend\" (2,000 users per arm is below the planned 3,000), control_rate is 12.0, variant_rate is 13.8 and lift_points is 1.8","The reason is one sentence that says the planned 3,000 users per arm was not reached, and it does not say to ship"]}