{"id":"evals3-judge-rubric-benefit-waiting-days","audience":"hr","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"format","title":"Write a three-rule rubric for judging one benefits field","free":false,"task":"Northgate Labs has a benefits chatbot, BenefitsBuddy, that fills a field called waiting_period_days. Write the AI judge's rubric for that one field as exactly 3 numbered rules, each under 20 words: one for correct, one for incorrect and one for unknown (unknown means the handbook gives no waiting period for that plan). Then add one last line showing the judge's output, exactly like {\"verdict\": \"...\", \"reason\": \"...\"}, with the reason limited to one sentence. Plain text, under 90 words. HR reviews the rubric before anyone uses it.","materialLabel":"Benefits handbook excerpt and bot samples (attached to your prompt automatically)","material":"NORTHGATE LABS BENEFITS HANDBOOK (excerpt, 2026)\nSilver medical plan: coverage starts after a 30-day waiting period.\nGold medical plan: coverage starts on the first of the month after 60 days of employment. (Waiting period: 60 days, then wait for the 1st.)\nDental plan: no waiting period applies to preventive care. Waiting periods for other dental care are not listed here; see the carrier brochure.\nVision plan: see carrier.\n\nBOT FIELD: waiting_period_days (a number, or null)\n\nSAMPLE BOT OUTPUTS\nA. Gold medical -> 60. Handbook says 60. (right)\nB. Silver medical -> 45. Handbook says 30. (wrong)\nC. Gold medical -> 90. Handbook says 60. (wrong)\nD. Vision -> 14. The handbook gives no number. (made up)\nE. Dental, major work -> 0. The handbook does not say. (made up)\n\nNOTES FROM PEOPLE OPS\n- The judge will look at one field per run. Tone, links and formatting are covered by other checks.\n- On earlier judges, \"close enough\" answers like 55 for 60 were being passed.\n- Do not put numbers about specific plans into the rubric; the same rubric runs on every plan.\n- The judge output is parsed by a script, so it can't wrap the JSON in other text.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Close enough counts","why":"The AI writes a soft rule like 'roughly matches', so 55 passes for 60.","fix":"Say: \"Correct only if the number exactly equals the handbook's number for that plan.\"","checks":["r0"]},{"title":"No unknown rule","why":"The AI writes only pass and fail, so a made-up number for a plan with no handbook figure can't be told apart from a wrong one.","fix":"Add: \"Unknown when the handbook gives no waiting period. Never guess.\"","checks":["r1"]},{"title":"A score or loose JSON","why":"The AI adds a 1-5 score or wraps the JSON in text.","fix":"Say: \"Last line: {verdict, reason} only, reason one sentence.\"","checks":["r2"]},{"title":"Wrong number of rules","why":"The AI adds extra rules about tone and links.","fix":"Say: \"Exactly 3 numbered rules, each under 20 words.\"","checks":["numberedItems"]},{"title":"Markdown and length","why":"AI defaults to bold labels, headings and long rules.","fix":"Ask for \"plain text, each rule under 20 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Asks for exactly 3 rules","patterns":["\\b(3|three)\\b.{0,25}rules?","exactly \\d+"]},{"label":"Names correct, incorrect and unknown","patterns":["unknown","incorrect"]},{"label":"Requires an exact match","patterns":["exact","equal","identical","no (rounding|close)"]},{"label":"Gives the JSON shape","patterns":["json","verdict.{0,30}reason"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good rubric rule is checkable: a number either matches or it doesn't. Spell out all three verdicts and say what unknown means so the judge never has to guess.","mission":"Write a judge rubric for one benefits field with unknown.","musts":["Exactly 3 numbered rules","One rule is unknown","Last line is JSON shape"],"rubricLabels":["Has exactly 3 numbered rules, one each for correct, incorrect and unknown, and the correct rule requires the bot's number to equal the handbook's number for the same plan exactly (no close enough)","The unknown rule applies when the handbook gives no waiting period for that plan, and tells the judge not to guess","The last line shows a JSON object with exactly the keys verdict and reason (no score, no other keys), and the reason is described as one sentence"]}