Library › AI Evals › Marketing and content

Write a JSON judge that checks ad prices against the list

Practice: Keeping the AI to your sources. About 5 minutes.

About this exercise

Your AI writes ad copy, and the costly failure is a price that doesn't match the price list. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 20 lines, with the placeholders {copy} and {price_list}. It returns only JSON {"pass": bool, "reason": str}; a false pass quotes the wrong price. Copy with no price passes, and '$49' and '$49.00' count as the same price. Under 40 words outside the block.

Common mistakes when you ask AI for this

  • One judge for everything. The AI also checks hype and footnotes, so a fail doesn't tell you a price was wrong. Try: Say: "Check only prices against the list."
  • Failing harmless copy. A strict judge fails ads with no price or flags $49.00 as different from $49. Try: Add: "No price passes. $49 and $49.00 are the same."
  • A score instead of a verdict. AI judges default to scores that a pipeline can't read. Try: Say: "Return only JSON with the keys pass and reason. A false pass quotes the wrong price."
  • An essay around the prompt. The AI explains the judge and adds examples in more code blocks. Try: Say: "Exactly 1 code block, at most 20 lines, one sentence after it."

What a strong prompt includes

  • Limits it to one failure mode
  • Handles the edge cases
  • Asks for JSON pass/reason
  • Uses placeholders
  • Limits the size

Practice this now ↑

Related exercises