{"id":"evals4-alert-or-not-two-standard-deviations","audience":"analysts","category":"evaluation","track":"evals","stage":"keep-honest","technique":"verification","level":"advanced","title":"Decide whether tonight's eval score should trigger an alert","free":false,"task":"You are on the analytics team watching Pedalbot's nightly refund-accuracy score. Apply the alert rule in the material to the last 7 nights and tonight's score, and show the math. Write plain text, exactly 6 lines labeled Average, Gaps squared, Standard deviation, Alert line, Tonight, Decision, each in the form 'Label: working = result' (for example 'Average: sum / 7 = result'). The Gaps squared line lists the squared gap for each night in the order given, then the total. Under 90 words.","materialLabel":"Nightly scores and the alert rule (attached to your prompt automatically)","material":"Pedalbot refund-accuracy eval - nightly score (percent of 200 cases passed)\n\nMon Nov 2   90\nTue Nov 3   93\nWed Nov 4   88\nThu Nov 5   91\nFri Nov 6   87\nSat Nov 7   92\nSun Nov 8   89\nMon Nov 9   86   <- tonight\n\nAlert rule (eval playbook, section 4): page the owner only if tonight's score is MORE than 2 standard deviations below the average of the previous 7 nights.\nStandard deviation here means: take each night's gap from the average, square it, average the squares (divide by 7), then take the square root. Do not use a 6-night version.\n\nContext: tonight is the lowest score of the week and 3 points lower than Sunday. The merge bar for the refund suite is 85 percent. No code changes shipped on Sunday or Monday.","rules":[{"type":"plainText"},{"type":"maxWords","max":90}],"promptHints":[{"label":"Uses the 7 previous nights","patterns":["previous 7","7 nights","seven nights","not tonight","excluding tonight"]},{"label":"Shows each step","patterns":["gaps squared","average","standard deviation","alert line"]},{"label":"Divides by 7","patterns":["divide by 7","/ ?7"]},{"label":"Fixes the line labels","patterns":["labeled","labels?","exactly 6","six lines"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown"]}],"mistakes":[{"title":"Alerting on the lowest score","why":"The AI sees the week's lowest score and says to alert.","fix":"Say: \"Alert only if MORE than 2 standard deviations below, so exactly 2 is not an alert.\"","checks":["r1"]},{"title":"Skipping the working","why":"The AI jumps to a verdict with no numbers.","fix":"Add: \"Show Average, Gaps squared, Standard deviation and Alert line.\"","checks":["r0"]},{"title":"Using the wrong divisor","why":"The AI divides by 6 or includes tonight in the average.","fix":"Add: \"Use the previous 7 nights only, divide by 7.\"","checks":["r0"]},{"title":"Extra commentary","why":"The AI adds bold headings and advice about the trend.","fix":"Add: \"Plain text, exactly 6 labeled lines, nothing else.\"","checks":["plainText","maxWords","r2"]}],"hint":"The rule says MORE than 2 standard deviations. Make the AI compute every step in order and name the label of every line so the edge case shows.","mission":"Apply the alert rule to tonight's score.","musts":["Use the previous 7 nights","Show every step","MORE than 2 SD"],"rubricLabels":["The math is right: Average 630 / 7 = 90.0; squared gaps in night order 0+9+4+1+9+4+1 = 28; Standard deviation sqrt(28 / 7) = 2.0; Alert line 90.0 - 2 x 2.0 = 86.0","The Tonight line says 86.0 is exactly 2.0 standard deviations below the average (not more than 2), and the Decision line says NO ALERT","There are exactly the 6 labeled lines in the order Average, Gaps squared, Standard deviation, Alert line, Tonight, Decision, with no intro or notes before or after them"]}