Library › AI Evals › Business analysts

Decide if an AI judge can be trusted against billing truth

  • For Business analysts
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Spokewise's AI judge grades whether Pedalbot quoted the right refund amount, and your code can check the true answer against the billing system. Write exactly 4 numbered lines: Disagreements (each id with truth then judge, like #9 Good/Bad), Agreement (with the math, like 6 of 10 = 60%), Bad answers caught (truth Bad that the judge also called Bad, with the math), and Verdict (trust or don't trust, compared to the 85% bar). Count a judge verdict of Unknown as a disagreement, and use any corrected truth. Plain text, under 85 words.

Common mistakes when you ask AI for this

  • Missing the correction. The AI reads the table and ignores the Nov 4 note, so #4 looks like a match. Try: Add: "Use the corrected truth for #4 from the notes."
  • Unknown treated as fine. The AI treats Unknown as a skip, not a disagreement, so agreement looks higher. Try: Say: "Count Unknown as a disagreement."
  • Hiding the math. The AI states a percent with no counts, so you can't check it. Try: Say: "Show the math, like 6 of 10 = 60%."
  • A soft verdict. The AI says the judge is 'mostly reliable' without comparing to your bar. Try: Say: "Verdict: trust or don't trust, compared to 85%."
  • Markdown and length. AI defaults to bold labels and a table, and runs long. Try: Ask for "plain text, each line under 20 words."

What a strong prompt includes

  • Uses the correction
  • Counts Unknown as a disagreement
  • Asks for the math
  • Gives the bar
  • Asks for plain text

Practice this now ↑

Related exercises