Library › AI Evals › Finance and accounting

Grade an AI's expense categories against current policy

Practice: Keeping the AI to your sources. About 5 minutes.

About this exercise

Your AP team is testing an AI that categorizes expenses, and you're grading 5 of its picks. Since Nov 1, software subscriptions under $50 a month go to Office Supplies, not Software. Write exactly 5 numbered lines: 'N. Correct' or 'N. Wrong - should be [category] - [the rule that applies]'. Plain text, under 90 words.

Common mistakes when you ask AI for this

  • Using the old guide. The guide still says all subscriptions are Software, and the AI doesn't know about the Nov 1 change. Try: Tell it: "Since Nov 1, subscriptions under $50 a month are Office Supplies."
  • No reason for a Wrong. The AI says a category is wrong without saying which rule decides it. Try: Ask: "For Wrong, give the right category and the rule."
  • Extra commentary. The AI adds notes about confidence scores and the attendee list. Try: Say: "Only the 5 lines, shaped N. Correct or N. Wrong - should be X - rule."
  • Getting markdown back. AI defaults to a bold table, which won't paste cleanly into your test log. Try: Ask for "plain text, no bold, no tables."

What a strong prompt includes

  • Gives the policy change
  • Shows the line shape
  • Asks for the rule
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises