{"id":"evals2-define-bar-from-cost-of-wrong","audience":"managers","category":"evaluation","track":"evals","stage":"define-success","level":"beginner","technique":"verification","title":"Set an accuracy bar per topic from the cost of a wrong answer","free":false,"task":"Spokewise accepts at most $600 a month of wrong-answer cost per topic. For each topic, work out the cost if every answer were wrong, the share of answers that may be wrong ($600 divided by that cost), and the accuracy bar. Write exactly 4 numbered lines like \"1. Hours: 6,000 x $2 = $12,000; $600 / $12,000 = 5%; bar 95%\", then a last line \"Strictest: [topic], [bar]\". Plain text, under 110 words.","materialLabel":"Pedalbot monthly volumes and costs (attached to your prompt automatically)","material":"Pedalbot - cost of a wrong answer, by topic (monthly)\n\nTopic | Chats per month | Cost of one wrong answer\n1. Hours | 6,000 | $2 (a customer's wasted trip, credited back)\n2. Prices | 2,000 | $15 (honoring a wrong quote)\n3. Refunds | 1,000 | $60 (a wrongly paid refund or a chargeback)\n4. Accident reports | 100 | $1,200 (claims handling and goodwill)\n\nBudget: Spokewise will accept at most $600 a month of wrong-answer cost per topic.\n\nBackground\n- Pedalbot's overall accuracy on the test set today is 97%.\n- The draft plan from last quarter used a single 95% bar for every topic.\n- Accuracy bar = 100% minus the share of answers that may be wrong.\n- Topics are listed in the order the team reviews them.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"One bar for everything","why":"The AI repeats the draft plan's single 95% bar.","fix":"Say: \"Work out a separate bar per topic from its own cost.\"","checks":["r2"]},{"title":"Skipping the working","why":"The AI states 98% without showing how it got there.","fix":"Add: \"Show the math on each line like 6,000 x $2 = $12,000.\"","checks":["r0","r1"]},{"title":"Mixing up the two steps","why":"The AI divides the wrong way or treats the $600 as per chat.","fix":"Say: \"Share allowed wrong = $600 / cost-if-all-wrong; bar = 100% minus that.\"","checks":["r1"]},{"title":"No final verdict line","why":"The AI ends with a paragraph of caveats.","fix":"Add: \"Last line: Strictest: [topic], [bar].\"","checks":["r2"]},{"title":"A report instead of lines","why":"The AI writes headings, bold and a long table.","fix":"Say: \"Exactly 4 numbered lines then one last line, plain text.\"","checks":["numberedItems","plainText","maxWords"]}],"promptHints":[{"label":"Asks for the math","patterns":["math","show","x \\$?\\d","divid"]},{"label":"Uses the $600 budget","patterns":["\\$600|budget"]},{"label":"Wants one bar per topic","patterns":["each topic|per topic|every topic|separate"]},{"label":"Gives a line format","patterns":["1\\.","bar \\d","like"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"A bar should be strict where a wrong answer is expensive. Have the AI do the same three steps for each topic and show them on the line.","mission":"Set an accuracy bar per topic from wrong-answer cost.","musts":["Math on every line","Separate bar per topic","Last line names strictest"],"rubricLabels":["Lines 1 and 2 show the math: Hours 6,000 x $2 = $12,000, $600 / $12,000 = 5%, bar 95%; Prices 2,000 x $15 = $30,000, $600 / $30,000 = 2%, bar 98%","Lines 3 and 4 show the math: Refunds 1,000 x $60 = $60,000, 1%, bar 99%; Accident reports 100 x $1,200 = $120,000, 0.5%, bar 99.5%","The last line starts \"Strictest:\" and names Accident reports with a bar of 99.5%, and no line uses one bar for every topic"]}