Library › AI Evals › Customer support teams

Rewrite the rubric line that made the judge pass wrong replies

  • For Customer support teams
  • You practice Showing an example of what you want
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Spokewise's AI judge passed three Pedalbot replies that a human marked as wrong. Find the one rubric line behind all three wrong verdicts and write exactly 3 labeled lines: 'Bad line:' (its number, like R4), 'Why:' (starts with "Because", under 20 words) and 'Rewrite:' (one replacement line under 30 words that starts with "Pass only if" and tells the judge to check the policy and ignore friendliness). Plain text, under 70 words.

Common mistakes when you ask AI for this

  • Blaming the wrong line. The AI points at the length or language rule because they look strict. Try: Say: "Check each judge reason against the rubric line it quotes."
  • A vague rewrite. The AI writes 'be accurate and helpful', which still lets a friendly wrong reply pass. Try: Say: "The rewrite must compare the reply's facts to the policy."
  • Rewriting the whole rubric. The AI hands back a new five-line rubric. Try: Say: "Replace only the one bad line, in 3 labeled lines."
  • Markdown and length. AI defaults to bold, bullets and a long explanation. Try: Ask for "plain text, under 70 words."

What a strong prompt includes

  • Asks for one line only
  • Points to the policy
  • Asks for the line number
  • Gives the labels
  • Asks for plain text

Practice this now ↑

Related exercises