{"id":"evals4-slack-alert-handoff-score-drop","audience":"product-managers","category":"evaluation","track":"evals","stage":"keep-honest","technique":"audience","level":"intermediate","title":"Write the on-call alert when the handoff score drops hard","free":false,"task":"Pedalbot's nightly handoff-accuracy score came in at 89 and the alert rule in the material applies. Write a 4-line plain-text Slack message to the on-call person. Line 1 is exactly 'ALERT: Pedalbot handoff score 89 tonight'. Line 2 is 'Math: avg A, SD S, alert line L, tonight N SD below' with real numbers. Line 3 is 'Check first: ' plus the deploy from the deploy log, as a thing to check and not a claimed cause. Line 4 is 'Owner: ' plus the on-call name. Under 55 words.","materialLabel":"Nightly scores and the alert rule (attached to your prompt automatically)","material":"Pedalbot handoff-accuracy eval - nightly score (percent of 200 cases passed)\n\nMon Nov 2   94\nTue Nov 3   95\nWed Nov 4   91\nThu Nov 5   97\nFri Nov 6   92\nSat Nov 7   96\nSun Nov 8   93\nMon Nov 9   89   <- tonight\n\nAlert rule (eval playbook, section 4): page the on-call owner if tonight's score is MORE than 2 standard deviations below the average of the previous 7 nights. Standard deviation: average the squared gaps from the average (divide by 7), then take the square root.\n\nOn-call this week: Priya Nair. Deploy log: Monday afternoon, handoff prompt v7 went live for 100 percent of riders. The team's merge bar for the handoff suite is 85 percent, so tonight's score is still above the bar.","rules":[{"type":"plainText"},{"type":"maxWords","max":55}],"promptHints":[{"label":"Fixes the four line openers","patterns":["alert:","math:","check first:","owner:"]},{"label":"Includes the numbers","patterns":["average","standard deviation","alert line"]},{"label":"Names who gets it","patterns":["priya","on-call","owner"]},{"label":"Treats the cause as a lead","patterns":["not (a )?confirmed","do not claim","don't claim","check"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown"]}],"mistakes":[{"title":"Excusing it with the bar","why":"The AI says 89 is still above 85, so no worry.","fix":"Say: \"The alert rule is standard deviations, not the merge bar.\"","checks":["r0"]},{"title":"Wrong statistics","why":"The AI uses the range or divides by 6 and gets the wrong threshold.","fix":"Add: \"Average and SD of the previous 7 nights, divide by 7.\"","checks":["r1"]},{"title":"Blaming v7 as fact","why":"The AI states the deploy caused the drop.","fix":"Add: \"Say 'check', do not claim the cause.\"","checks":["r2"]},{"title":"Long Slack post","why":"The AI writes greetings, bullets and bold.","fix":"Add: \"Plain text, exactly 4 lines, under 55 words.\"","checks":["plainText","maxWords"]}],"hint":"An alert message is for one reader who must act. Give the line openers, make the math visible, and say the cause is only something to check.","mission":"Write the on-call alert with the math.","musts":["Four lines, fixed openers","Show the 7-night math","Cause is a lead only"],"rubricLabels":["Line 1 is exactly 'ALERT: Pedalbot handoff score 89 tonight', and the message does not say 'no alert' or 'within normal range' because 89 is above the 85 percent bar","Line 2 is exactly 'Math: avg 94.0, SD 2.0, alert line 90.0, tonight 2.5 SD below'","Line 3 begins 'Check first:' and names the Monday handoff prompt v7 deploy without claiming it caused the drop, and line 4 is exactly 'Owner: Priya Nair'"]}