Library › AI Evals › Software developers

Write an AI-judge prompt that grades one field with an unknown option

  • For Software developers
  • You practice Setting limits and rules
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Pedalbot, the support chatbot of Spokewise, returns JSON with several fields, and you are building an AI judge for one field only: late_fee_stated. Write the judge prompt in exactly 1 code block of at most 22 lines, using the placeholders {policy}, {question} and {late_fee_stated}. The judge must return only JSON with verdict (exactly correct, incorrect or unknown) and reason (one sentence). Use unknown when the policy text does not give the fee being asked about. Keep text outside the code block under 30 words.

Common mistakes when you ask AI for this

  • Grading the whole reply. The AI writes a judge that scores the whole answer, tone included, which is not the one field you need. Try: Say: "Judge only late_fee_stated. Ignore tone, answer text and station."
  • No way to say unknown. Without an escape hatch the judge must call a made-up cargo fee either right or wrong. Try: Add: "If the policy does not state the fee, the verdict is unknown. Do not guess."
  • A score or loose text. AI judges default to 1-10 scores or a paragraph of explanation, which a script cannot parse. Try: Say: "Output only JSON with exactly verdict and reason, reason one sentence."
  • Several code blocks. The AI adds usage notes and extra examples in more code blocks. Try: Say: "Exactly 1 code block, at most 22 lines, under 30 words outside it."

What a strong prompt includes

  • Limits the judge to one field
  • Adds an unknown verdict
  • Asks for JSON verdict and reason
  • Uses the placeholders
  • Limits the size

Practice this now ↑

Related exercises