Library › AI Evals › Managers and team leads

Plan how much live traffic to grade and who audits the judge

  • For Managers and team leads
  • You practice Setting limits and rules
  • Time about 5 minutes
  • Level Advanced

About this exercise

You run Pedalbot's quality program. Plan the weekly live-traffic grading within the budgets in the material. Write exactly 4 plain-text lines starting SHARE:, JUDGE GRADES:, AUDIT:, BAR:. SHARE shows the dollar math, the number of conversations and the percent of weekly traffic, and says they are picked at random. JUDGE GRADES names the three checks, each pass or fail. AUDIT names the person, how many grades they check, and the pass/fail split. BAR gives the agreement bar and what happens below it. Under 95 words.

Common mistakes when you ask AI for this

  • Grading everything. The AI goes along with 'more is better' and ignores the budget. Try: Say: "Stay inside the $200 weekly budget; show the math."
  • Adding checks the judge doesn't do. The AI lists price accuracy and spelling too. Try: Add: "Only the three judge checks in the material."
  • Auditing only failures. The AI has the human review only what the judge failed. Try: Add: "Audit half judge-pass and half judge-fail."
  • Long plan with headings. The AI writes a bold-headed plan. Try: Add: "Exactly 4 plain lines starting SHARE:, JUDGE GRADES:, AUDIT:, BAR:."

What a strong prompt includes

  • Shows budget math
  • Fixes the four labels
  • Splits the audit
  • Names the person
  • Asks for plain text

Practice this now ↑

Related exercises