Library › AI Evals › Customer support teams

Grade 4 bot refund replies against your rubric

Practice: Asking for a format you can use. About 5 minutes.

About this exercise

Your support lead wants this week's bot spot-check graded in a format she can paste into the QA sheet. Grade the 4 replies against the 3 criteria in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C2 - "exact words from the reply"', naming the failed criterion and quoting the words that fail it. Use the refund policy that is current now. Plain text, under 90 words, nothing else.

Common mistakes when you ask AI for this

  • Grading with the old policy. The policy lists both 14 and 30 days, and the AI may pass Reply 3 because it matches the old window. Try: Add: "Use the current 30-day window. 14 days is out of date."
  • No evidence behind a FAIL. The AI writes 'fails C2' without showing why, so your lead can't check the grade. Try: Say: "Quote the exact words from the reply that fail it."
  • Paragraphs instead of lines. By default the AI writes a section per reply with a verdict for each criterion. Try: Show the shape: "1. FAIL - C2 - \"quote\"", one line per reply."
  • Getting markdown back. AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker. Try: Ask for "plain text, no bold, no headings, no tables."
  • Trusting the AI to count words. AI is bad at counting. Ask for a total and it often lands 20% over. Try: Cap each line instead: "each line under 25 words."

What a strong prompt includes

  • Shows the line shape
  • Asks for a quote as evidence
  • Flags the current policy
  • Says how many lines
  • Asks for plain text

Practice this now ↑

Related exercises