{"id":"eval-hr-golden-answer-pto-carryover","audience":"hr","category":"evaluation","track":"evals","technique":"context","title":"Write the golden answer for an HR bot's PTO question","free":false,"task":"You're building test cases for the HR policy bot and need a golden answer for a PTO question. For 2027, the carryover limit drops from 40 to 24 hours (approved yesterday, not in the handbook yet). Write exactly 3 numbered lines: '1. Answer: ...' (direct), '2. Math: ...' (the employee's hours worked out), and '3. Next step: ...'. Use only the handbook plus that change. Plain text, under 50 words.","materialLabel":"The test question and the handbook section (attached to your prompt automatically)","material":"TEST CASE #9 for PolicyPal (Ostrander Freight HR bot)\nEmployee question (asked Dec 2): \"I have 56 hours of PTO left. How much can I carry over into 2027, and what happens to the rest?\"\n\nHANDBOOK - Section 6.3, Paid time off (2026 edition)\n- Employees may carry over up to 40 hours of unused PTO into the next calendar year.\n- Unused PTO above the carryover limit is forfeited on Dec 31. It is not paid out.\n- Employees can check their balance in the Workday app under Time Off.\n- Questions about a specific balance go to the HR service desk (hr-help@ostrander.example).\n\nTest-writing notes\n- PolicyPal's last answers were 150+ words and suggested \"using it or losing it\" with vacation ideas.\n- Golden answers are the shortest complete correct answer we compare PolicyPal to.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":50}],"mistakes":[{"title":"Using the old limit","why":"The handbook still says 40 hours, and the AI doesn't know about yesterday's change.","fix":"Tell it: \"For 2027 the carryover limit is 24 hours.\"","checks":["r0","r1"]},{"title":"Hiding the math","why":"The AI says 'you'll lose some hours' without the number.","fix":"Ask: \"Show the math: 56 - 24 = hours lost.\"","checks":["r1"]},{"title":"Friendly extras","why":"The AI adds vacation tips and 'use it or lose it' advice.","fix":"Say: \"Use only the handbook. End with a next step from it.\"","checks":["r2"]},{"title":"A long answer","why":"The AI writes a paragraph, which is hard to grade against.","fix":"Show the shape: \"1. Answer: 2. Math: 3. Next step:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives the new limit","patterns":["24","2027","new limit|changed"]},{"label":"Shows the three-line shape","patterns":["answer:","math:","next step"]},{"label":"Asks for the math","patterns":["math","56","show"]},{"label":"Keeps to the handbook","patterns":["only.{0,30}(handbook|policy)","no (vacation|tips|extras)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer has to reflect the current rule, even before the handbook catches up. Give the AI the change, show the exact shape, and ask for the math.","rubricLabels":["Uses the new 24-hour limit (not 40): the employee can carry over 24 hours","The Math line shows 56 - 24 = 32 hours forfeited on Dec 31 and not paid out","Line 3 points to checking the balance in Workday or contacting hr-help@ostrander.example, and the answer adds no vacation ideas or anything not in the handbook"]}