Library › AI Evals › HR and recruiting

Write a three-rule rubric for judging one benefits field

  • For HR and recruiting
  • You practice Asking for a format you can use
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Northgate Labs has a benefits chatbot, BenefitsBuddy, that fills a field called waiting_period_days. Write the AI judge's rubric for that one field as exactly 3 numbered rules, each under 20 words: one for correct, one for incorrect and one for unknown (unknown means the handbook gives no waiting period for that plan). Then add one last line showing the judge's output, exactly like {"verdict": "...", "reason": "..."}, with the reason limited to one sentence. Plain text, under 90 words. HR reviews the rubric before anyone uses it.

Common mistakes when you ask AI for this

  • Close enough counts. The AI writes a soft rule like 'roughly matches', so 55 passes for 60. Try: Say: "Correct only if the number exactly equals the handbook's number for that plan."
  • No unknown rule. The AI writes only pass and fail, so a made-up number for a plan with no handbook figure can't be told apart from a wrong one. Try: Add: "Unknown when the handbook gives no waiting period. Never guess."
  • A score or loose JSON. The AI adds a 1-5 score or wraps the JSON in text. Try: Say: "Last line: {verdict, reason} only, reason one sentence."
  • Wrong number of rules. The AI adds extra rules about tone and links. Try: Say: "Exactly 3 numbered rules, each under 20 words."
  • Markdown and length. AI defaults to bold labels, headings and long rules. Try: Ask for "plain text, each rule under 20 words."

What a strong prompt includes

  • Asks for exactly 3 rules
  • Names correct, incorrect and unknown
  • Requires an exact match
  • Gives the JSON shape
  • Asks for plain text

Practice this now ↑

Related exercises