Library › AI Evals › Teachers

Check if an AI grader agrees with you on short answers

Practice: Catching mistakes and made-up facts. About 5 minutes.

About this exercise

You let an AI grade 10 short answers you already graded, to see if it can score practice quizzes. Your department allows it only with at least 80% agreement and at most 1 false credit (AI gives credit when you didn't). Write exactly 4 numbered lines: Disagreements (ids, both grades), Agreement (the math), False credits (count and ids), and Decision (Use for practice quizzes or Don't use yet, with the reason). Plain text, under 80 words.

Common mistakes when you ask AI for this

  • Only checking agreement. The AI reports a percentage but doesn't separate the dangerous misses, where students get credit for a wrong idea. Try: Add: "Count false credits: AI Credit when I gave No credit."
  • Hiding the math. The AI says it 'mostly agrees' without counts. Try: Ask: "Show the math, like 10 - 3 = 7 of 10 = 70%."
  • No clear call. The AI says the grader is 'promising' instead of applying your department's rules. Try: Say: "Decision: Use for practice quizzes or Don't use yet, using 80% and at most 1 false credit."
  • A long analysis. The AI writes about AI grading in general. Try: Be exact: "4 numbered lines: Disagreements, Agreement, False credits, Decision."

What a strong prompt includes

  • Checks false credits
  • Asks for the math
  • Gives the rules
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises