{"id":"eval-mgr-team-update-vs-project-tracker","audience":"managers","category":"evaluation","track":"evals","technique":"verification","title":"Check an AI team update against your project tracker","free":false,"task":"An AI drafted your weekly team update from the project tracker, and you want to check it before it goes to your VP. Write exactly 5 numbered lines, one per sentence, each labeled Matches, Wrong or Not in tracker. For each Wrong line, add the tracker's version in quotation marks. Judge only by the tracker. Plain text, under 100 words.","materialLabel":"The tracker export and the AI's draft (attached to your prompt automatically)","material":"Data Platform team - tracker export (Nov 6)\n\nDP-14 Warehouse migration. Status: In progress, 60% done. Due Dec 12. Owner: Kai.\nDP-19 Cost dashboard v2. Status: Done (shipped Nov 4). Owner: Lena.\nDP-22 Access review automation. Status: Blocked, waiting on security sign-off. Owner: Ravi.\nDP-25 Pipeline alerting. Status: Not started. Due Jan 15. Owner: Kai.\nDP-27 Data catalog pilot. Status: In progress, 30% done. Owner: Lena.\n\nAI-DRAFTED UPDATE (sentences numbered by you)\n(1) The cost dashboard v2 shipped on Nov 4. (2) The warehouse migration is 80% done and on track for Dec 12. (3) Access review automation is blocked on security sign-off. (4) Pipeline alerting is in progress. (5) The team also finished onboarding two new analysts this week.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Asking 'is it ok?'","why":"The AI says the update reads well and misses the changed percentage.","fix":"Say: \"Label each sentence Matches, Wrong or Not in tracker.\"","checks":["r0","r2"]},{"title":"Letting extras through","why":"The onboarding news sounds plausible, so the AI accepts it.","fix":"Add: \"Not in tracker means no tracker item mentions it.\"","checks":["r0"]},{"title":"No proof for corrections","why":"The AI says a status is off without showing the tracker's version.","fix":"Add: \"For Wrong, quote the tracker.\"","checks":["r1"]},{"title":"A rewritten update","why":"The AI 'helps' by rewriting the update for your VP.","fix":"Be exact: \"Exactly 5 numbered lines, one per sentence, nothing else.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives exact labels","patterns":["matches","wrong","not in tracker","label"]},{"label":"Makes the tracker the judge","patterns":["only.{0,30}(tracker|export)","against the tracker"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|sentence)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI updates round progress up and add news that isn't in your data. Give the AI three labels and make the tracker the only judge.","rubricLabels":["Labels sentences 1 and 3 Matches, sentences 2 and 4 Wrong, and sentence 5 Not in tracker","The Wrong lines give the tracker's version in quotation marks: 60% done (not 80%) and Not started (not in progress)","Uses only the three labels Matches, Wrong and Not in tracker, with no rewritten update or advice"]}