{"id":"evals4-capstone-eval-plan-bike-size-helper","audience":"product-managers","category":"evaluation","track":"evals","stage":"keep-honest","technique":"format","level":"advanced","title":"Write a one-page eval plan for a new Pedalbot feature","free":false,"task":"Spokewise is launching Pedalbot's bike-size helper. Write a one-page eval plan in plain text using exactly these five heading lines, in order: Cases, Graders, Accuracy bar, What runs on every change, Live sampling. Under each heading write exactly 2 lines starting '- '. Use these facts: the accuracy bar is 95% overall with 0 failures on helmet rules; the every-change set is 12 cases; live sampling is 5% of the weekly chats (say the number of chats) and the support lead audits the judge. Under 300 words.","materialLabel":"Feature spec (attached to your prompt automatically)","material":"Pedalbot bike-size helper - feature spec v2 (Spokewise, launch Dec 1)\n\nWhat it does: the rider gives their height in cm and how they plan to ride. Pedalbot names a bike size and reminds them to take a helmet.\n\nSize chart (the only allowed answers)\n- Under 150 cm: XS\n- 150 to 164 cm: S\n- 165 to 174 cm: M\n- 175 to 184 cm: L\n- 185 cm and over: XL\n- Under 140 cm: no adult bike. Pedalbot hands off to a store to see kids' bikes.\n\nHelmet rules: a helmet is required on every rental. Pedalbot never says it is optional, even if the rider asks, and always mentions it in a size answer.\n\nTraffic: about 3,000 size chats a week expected. Riders sometimes give height in feet and inches (\"5'9\") or say \"between M and L\".\n\nBeta notes (40 chats): 3 answers gave a size one step off at a boundary height. 2 riders said \"helmet is optional, right?\" and the bot agreed once. The tone judge has been reliable on friendliness but has missed wrong sizes.\n\nOpen question from the product owner: \"Can we just have the AI judge grade everything?\" ","rules":[{"type":"plainText"},{"type":"maxBullets","max":10},{"type":"maxWords","max":300}],"promptHints":[{"label":"Fixes the five headings","patterns":["cases","graders","accuracy bar","every change","live sampling"]},{"label":"Asks for 2 lines per heading","patterns":["2 (lines|bullets)","two (lines|bullets)","exactly 2"]},{"label":"Splits code check from judge","patterns":["code check","exact","lookup","chart"]},{"label":"Names edge cases","patterns":["boundary","under 140","helmet"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown"]}],"mistakes":[{"title":"Judge grades everything","why":"The AI uses an AI judge for sizes too, though a chart lookup is exact.","fix":"Say: \"Grade sizes with a code check against the chart; use the AI judge only for tone.\"","checks":["r1"]},{"title":"Only happy-path cases","why":"The AI lists ordinary heights and skips boundaries, kids and helmet pushback.","fix":"Add: \"Include a boundary height, an under-140 rider, and a helmet-optional pushback.\"","checks":["r1"]},{"title":"One blended bar","why":"The AI gives one accuracy target and no zero-tolerance rule.","fix":"Add: \"95% overall, and 0 failures on helmet rules.\"","checks":["r2"]},{"title":"Skipping the audit owner","why":"The AI samples live chats but never says who checks the judge.","fix":"Add: \"Name who audits the judge and the number of chats.\"","checks":["r2"]},{"title":"Essay with markdown","why":"The AI writes headings with # and long bullets.","fix":"Add: \"Plain text, five heading lines, 2 dash lines each, under 240 words.\"","checks":["plainText","maxBullets","maxWords","r0"]}],"hint":"An eval plan answers five questions in order. Name the headings, give each a fixed size, and say which checks a person or code can do better than an AI judge.","mission":"Write a one-page eval plan for the new feature.","musts":["Five headings, 2 lines each","Code check for sizes","Name the judge auditor"],"rubricLabels":["The five heading lines appear exactly as given and in order (Cases, Graders, Accuracy bar, What runs on every change, Live sampling), each with exactly 2 lines starting '- '","Cases include a boundary height (164 or 165, 174 or 175, 184 or 185 cm), a rider under 140 cm handed off to a store, and a rider pushing back that a helmet is optional; Graders gives the size check to a code or exact-match lookup against the chart and the tone check to an AI judge","Accuracy bar says 95% overall and 0 helmet-rule failures; What runs on every change names a 12-case set; Live sampling says 5% of 3,000 = 150 chats a week and names the support lead as the auditor"]}