{"id":"eval-fin-variance-commentary-vs-actuals","audience":"finance","category":"evaluation","track":"evals","technique":"verification","title":"Check AI variance commentary against the actual numbers","free":false,"task":"An AI wrote the monthly variance commentary for your budget review, and every number must match the report before it goes to the CFO. Write exactly 5 numbered lines, one per sentence, each labeled Matches, Wrong or Not in report. For each Wrong line, add the report's figure in quotation marks. Judge only by the report. Plain text, under 100 words.","materialLabel":"The budget vs actual report and the AI's commentary (attached to your prompt automatically)","material":"Thornbury Health Services - October budget vs actual (operations, $ thousands)\n\nLine | Budget | Actual | Variance\nSalaries | 820 | 845 | 25 over\nContract labor | 60 | 112 | 52 over\nSupplies | 140 | 128 | 12 under\nRent | 95 | 95 | 0\nSoftware | 38 | 51 | 13 over\nTotal | 1,153 | 1,231 | 78 over\n\nReport note: contract labor overage is from covering two open roles in billing.\n\nAI-DRAFTED COMMENTARY (sentences numbered by you)\n(1) Total operating spend came in $78K over budget for October. (2) Contract labor drove most of the overage, running $52K over budget to cover two open billing roles. (3) Supplies were $21K under budget. (4) Software spend was $13K over, driven by a new analytics license. (5) Rent was on budget.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Asking for a review","why":"The AI polishes the wording and never checks each number against the report.","fix":"Say: \"Label each sentence Matches, Wrong or Not in report.\"","checks":["r0","r2"]},{"title":"Letting an invented reason pass","why":"The $13K is right, so the AI misses that the 'analytics license' reason isn't in the report.","fix":"Add: \"Not in report includes reasons the report doesn't give.\"","checks":["r0"]},{"title":"No proof for corrections","why":"The AI says a number is off without giving the report's figure.","fix":"Add: \"For Wrong, quote the report's figure.\"","checks":["r1"]},{"title":"A rewritten commentary","why":"The AI 'helps' by rewriting the paragraph for the CFO.","fix":"Be exact: \"Exactly 5 numbered lines, one per sentence, nothing else.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives exact labels","patterns":["matches","wrong","not in report","label"]},{"label":"Makes the report the judge","patterns":["only.{0,30}(report|table)","against the (report|table|numbers)"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|sentence)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI commentary often gets one number wrong or invents a reason for a variance. Give the AI three labels, make the report the only judge, and ask for the real figure when it's wrong.","rubricLabels":["Labels sentences 1, 2 and 5 Matches, sentence 3 Wrong, and sentence 4 Not in report (the report gives the $13K but not the analytics license reason)","The Wrong line for sentence 3 gives the report's figure of 12 (thousand) under budget in quotation marks","Uses only the three labels Matches, Wrong and Not in report, with no rewritten commentary or extra analysis"]}