{"id":"evals4-what-runs-every-change-nightly","audience":"managers","category":"evaluation","track":"evals","stage":"keep-honest","technique":"constraints","level":"intermediate","title":"Sort 8 eval jobs into every change, nightly, or never waits","free":false,"task":"You manage Pedalbot's eval pipeline. Every-change jobs must each finish in under 2 minutes and cost under $1 together. Anything that checks a reply before a rider sees it, or alerts on live traffic, never waits on a batch. Everything else runs nightly. Write plain text with three heading lines exactly 'EVERY CHANGE', 'NIGHTLY', 'NEVER WAITS ON A BATCH', each followed by '- Job name: reason under 8 words' lines, using every job once. End with the line 'Every-change cost: $X + $Y = $Z'. Under 190 words.","materialLabel":"Eval jobs (attached to your prompt automatically)","material":"Spokewise Pedalbot - eval jobs and what they cost (Nov 2026)\n\n1. Smoke set - 12 cases, takes 40 seconds, $0.30 per run.\n2. Full regression - 400 cases, takes 25 minutes, $9.00 per run.\n3. Tone judge set - 150 cases graded by an AI judge, takes 40 minutes, $14.00 per run.\n4. Adversarial set - 80 prompt-injection and jailbreak cases, takes 15 minutes, $5.00 per run.\n5. Injury-safety set - 20 cases about crashes and injured riders, takes 90 seconds, $0.60 per run.\n6. Live-sample grading - 300 of yesterday's real conversations graded by the judge, takes 60 minutes, $20.00 per run.\n7. Live reply filter - checks every reply before the rider sees it, adds 0.2 cents per reply.\n8. Burst alert - pages the on-call person if the live filter blocks 5 or more replies within 10 minutes.\n\nContext: about 14 code changes land per weekday. Last quarter the team once waited until the next morning's batch to learn a change had made Pedalbot ignore an injury report. The adversarial set has 6 cases that overlap with the smoke set.","rules":[{"type":"plainText"},{"type":"maxWords","max":190}],"promptHints":[{"label":"States the time limit","patterns":["2 minutes","two minutes","under 2"]},{"label":"States the cost limit","patterns":["\\$1","under \\$?1","one dollar"]},{"label":"Names the three groups","patterns":["every change","nightly","never waits"]},{"label":"Uses every job once","patterns":["every job","all 8","each job","once"]},{"label":"Asks for the cost sum line","patterns":["every-change cost","\\$0\\.30","add up|sum|total"]}],"mistakes":[{"title":"Running everything every change","why":"The AI puts the full regression on every change because more checking sounds safer.","fix":"Say: \"Every-change jobs must be under 2 minutes each and under $1 together.\"","checks":["r0","r1"]},{"title":"Treating live checks as batch","why":"The AI schedules the live filter and burst alert nightly.","fix":"Add: \"Anything checking a reply before the rider sees it, or alerting on live traffic, never waits on a batch.\"","checks":["r1"]},{"title":"Missing the cost line","why":"The AI never adds up the every-change cost.","fix":"Add: \"End with: Every-change cost: $X + $Y = $Z.\"","checks":["r2"]},{"title":"Long explanations","why":"The AI writes a paragraph per job under markdown headings.","fix":"Add: \"Plain text, reason under 8 words per job.\"","checks":["plainText","maxWords"]}],"hint":"Group by the rule you were given: time and cost for every change, timing of the check for never-waits. Ask for the cost sum so the budget is visible.","mission":"Sort eval jobs by when they must run.","musts":["Three groups, every job once","Respect time and cost limit","End with cost sum"],"rubricLabels":["Every-change holds exactly the Smoke set and the Injury-safety set (the only jobs under 2 minutes each)","Nightly holds Full regression, Tone judge set, Adversarial set and Live-sample grading; Never waits on a batch holds the Live reply filter and the Burst alert; all 8 jobs appear exactly once","The three headings are written exactly as given, each job line is '- Job name: reason', and the last line is exactly 'Every-change cost: $0.30 + $0.60 = $0.90'"]}