{"id":"evals4-plan-live-traffic-sample","audience":"managers","category":"evaluation","track":"evals","stage":"keep-honest","technique":"constraints","level":"advanced","title":"Plan how much live traffic to grade and who audits the judge","free":false,"task":"You run Pedalbot's quality program. Plan the weekly live-traffic grading within the budgets in the material. Write exactly 4 plain-text lines starting SHARE:, JUDGE GRADES:, AUDIT:, BAR:. SHARE shows the dollar math, the number of conversations and the percent of weekly traffic, and says they are picked at random. JUDGE GRADES names the three checks, each pass or fail. AUDIT names the person, how many grades they check, and the pass/fail split. BAR gives the agreement bar and what happens below it. Under 95 words.","materialLabel":"Budgets and constraints (attached to your prompt automatically)","material":"Pedalbot live-traffic quality plan - inputs (Nov 2026)\n\nVolume: about 20,000 conversations a week.\nAI judge cost: $0.04 per conversation graded. Weekly judge budget: $200.\nJudge checks available: policy match (did the reply follow the rental policy), tone (calm and polite), handoff (did it pass the rider to a human when required). Price accuracy and spelling are checked elsewhere.\n\nHuman audit: Marta Okafor, support lead, has 5 hours a week for audits and reviews about 10 conversations an hour.\n\nData team suggestion (Nov 3): \"Grade all 20,000. More is better.\" Finance reply: weekly judge budget is fixed at $200.\n\nLast quarter the judge passed 12 replies a human later found wrong, and failed 9 replies a human found fine, in a spot check of 60.\n\nAudit idea from Marta: look at the judge's grades, half the ones it passed and half the ones it failed, so both kinds of mistakes show up.\nTeam rule: if the judge and the human agree on fewer than 90 percent of audited grades, retune the judge before trusting its scores.","rules":[{"type":"plainText"},{"type":"maxWords","max":95}],"promptHints":[{"label":"Shows budget math","patterns":["budget","\\$200","\\$0\\.04","math"]},{"label":"Fixes the four labels","patterns":["share:","judge grades:","audit:","bar:"]},{"label":"Splits the audit","patterns":["half","25","pass and","pass/fail"]},{"label":"Names the person","patterns":["marta","named person","support lead"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown"]}],"mistakes":[{"title":"Grading everything","why":"The AI goes along with 'more is better' and ignores the budget.","fix":"Say: \"Stay inside the $200 weekly budget; show the math.\"","checks":["r0"]},{"title":"Adding checks the judge doesn't do","why":"The AI lists price accuracy and spelling too.","fix":"Add: \"Only the three judge checks in the material.\"","checks":["r1"]},{"title":"Auditing only failures","why":"The AI has the human review only what the judge failed.","fix":"Add: \"Audit half judge-pass and half judge-fail.\"","checks":["r2"]},{"title":"Long plan with headings","why":"The AI writes a bold-headed plan.","fix":"Add: \"Exactly 4 plain lines starting SHARE:, JUDGE GRADES:, AUDIT:, BAR:.\"","checks":["plainText","maxWords"]}],"hint":"Three questions, three answers: how much, graded on what, checked by whom. Use the budgets as hard limits and make the audit see both kinds of judge mistakes.","mission":"Plan the live sample and who audits the judge.","musts":["Stay inside budget","Judge names three checks","Audit both pass and fail"],"rubricLabels":["The line starting 'SHARE:' shows the right math ($200 / $0.04 = 5,000 conversations = 25% of 20,000), says random, and does not propose grading all 20,000","The line starting 'JUDGE GRADES:' is exactly 'JUDGE GRADES: policy match, tone, handoff - each pass or fail' with no other checks added","The line starting 'AUDIT:' names Marta Okafor checking 50 grades (5 hours x 10 an hour) split 25 judge-pass and 25 judge-fail, and the line starting 'BAR:' says 45 of 50 (90%) must agree, else retune the judge before trusting its scores"]}