Library › AI Evals › Software developers

Write a JSON judge that catches made-up SQL columns

Practice: Setting limits and rules. About 5 minutes.

About this exercise

Your text-to-SQL assistant sometimes queries columns that don't exist, and you want an LLM judge for that one failure mode only, not speed or correctness. Write the judge prompt in exactly 1 code block of at most 30 lines, using the placeholders {schema} and {sql}. It must make the judge return only JSON like {"pass": true, "reason": "..."}, and when pass is false the reason must name the unknown column or table. Keep text outside the code block under 40 words.

Common mistakes when you ask AI for this

  • One judge for everything. The AI happily writes a judge that checks every failure mode at once, which makes its verdicts hard to trust or debug. Try: Say: "Judge only unknown columns or tables. Ignore speed, dates and style."
  • A score instead of a verdict. AI judges default to 1-10 scores, which drift and can't gate a build. Try: Say: "Return only JSON with two keys: pass (true/false) and reason."
  • A reason that says nothing. Without guidance the judge writes reasons like 'query has issues'. Try: Add: "If pass is false, the reason names the unknown column or table."
  • Forgetting the placeholders. The AI pastes this week's schema into the prompt, so the judge breaks when the schema changes. Try: Say: "Use {schema} and {sql} as placeholders."
  • An essay around the prompt. The AI explains how judges work and adds usage tips and extra examples in more code blocks. Try: Say: "Exactly 1 code block, at most 30 lines, under 30 words outside it."

What a strong prompt includes

  • Limits it to one failure mode
  • Asks for JSON pass/reason
  • Uses placeholders
  • Makes the reason name the column
  • Limits the size

Practice this now ↑

Related exercises