{"id":"eval-off-golden-answer-password-reset-portal","audience":"office","category":"evaluation","track":"evals","technique":"context","title":"Write the golden answer for an IT helpdesk bot question","free":false,"task":"You're the office manager writing test cases for the IT helpdesk bot, and this password question needs a golden answer. Since Monday, password resets use the new portal at reset.calderwebb.example, where you sign in with your badge number and a code texted to your work phone, and the old phone line is closed. Write exactly 3 numbered lines: '1. Where: ...', '2. Steps: ...' and '3. If locked out: ...'. Use only the IT page plus that change. Plain text, under 50 words.","materialLabel":"The test question and the IT help page (attached to your prompt automatically)","material":"TEST CASE #6 for the Calder & Webb IT helpdesk bot\nEmployee question: \"I forgot my computer password. How do I reset it?\"\n\nIT HELP PAGE: \"Resetting your password\" (last updated in August)\n- Call the IT reset line at ext. 4400 and verify your employee ID.\n- Or ask your office manager to open a ticket for you.\n- New passwords need at least 12 characters and can't reuse your last 5.\n- After 5 wrong tries your account locks for 30 minutes. If it's still locked after that, visit the IT desk on floor 2.\n\nTest-writing notes\n- The bot's answers in testing were long and sometimes told people to email their password to IT, which we never want.\n- Golden answers are the shortest complete correct answer we grade the bot against.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":50}],"mistakes":[{"title":"Using the closed phone line","why":"The help page still lists ext. 4400, and the AI doesn't know it closed Monday.","fix":"Tell it: \"Resets now use reset.calderwebb.example. The phone line is closed.\"","checks":["r0"]},{"title":"Old steps for a new portal","why":"The AI doesn't know how the new portal works, so it gives the old phone steps.","fix":"Add: \"Steps: sign in with your badge number and the texted code.\"","checks":["r1"]},{"title":"No lockout path","why":"The AI forgets what to do if the account locks.","fix":"Add: \"If locked out: the 30-minute lock and the IT desk on floor 2.\"","checks":["r2"]},{"title":"A long reply","why":"The AI writes a paragraph that's hard to grade against.","fix":"Show the shape: \"1. Where: 2. Steps: 3. If locked out:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives the new portal","patterns":["reset\\.calderwebb","portal","since monday","closed"]},{"label":"Shows the three-line shape","patterns":["where:","steps:","locked out"]},{"label":"Gives the new portal steps","patterns":["badge","text(ed)? code","work phone"]},{"label":"Keeps to the IT page","patterns":["only.{0,30}(it page|help page|page)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer has to match how things work today, not what the old help page says. Give the AI the change, show the exact shape, and include the lockout path.","rubricLabels":["Line 1 points to the new portal at reset.calderwebb.example and does not send them to the ext. 4400 phone line","Line 2 says to sign in with your badge number and the code texted to your work phone","Line 3 says the account locks for 30 minutes after 5 wrong tries and to visit the IT desk on floor 2 if still locked, and the answer never suggests sharing or emailing a password"]}