Library › AI Evals › Finance and accounting

Choose between two judges when one grades its own model's work

  • For Finance and accounting
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Brightfield Finance Ops uses an AI model to summarize vendor invoices, and two AI judges each graded the same 20 summaries. Humans also checked the 20. Output a markdown table with one row per judge and the columns Judge, Passed, Pass rate and Gap vs humans (in points), then one last line starting "Recommendation:" that picks a judge in under 20 words. No other text. Under 70 words.

Common mistakes when you ask AI for this

  • Missing the human baseline. The AI compares the two judges to each other and never to the human 70%. Try: Say: "Gap vs humans = judge pass rate minus 14 of 20 = 70%."
  • Choosing on cost. The AI picks the cheaper, faster judge because the notes mention it. Try: Say: "Choose on whether the judge grades its own model's work."
  • Calling the summaries solved. The AI repeats the 19 of 20 headline. Try: Add: "Compare each judge to the human result."
  • Extra text and bold. AI adds an intro, bold and notes. Try: Say: "Table, then one Recommendation line, plain text, nothing else."

What a strong prompt includes

  • Compares to the human result
  • Asks for points
  • Asks for a table
  • Gives the check for self-grading
  • Asks for one Recommendation line

Practice this now ↑

Related exercises