Library › AI Evals › Product managers

Write a 5-line pass/fail rubric for an AI meeting recap

Practice: Setting limits and rules. About 5 minutes.

About this exercise

You own the AI Recap feature at Tallow Notes and need a rubric your team can grade recaps with. Write exactly 5 numbered criteria, each one line in the form 'Pass if ...; Fail if ...', binary only (no scores or scales). The criteria must cover the current word limit, action items with no named owner, and the paused part of the meeting. Plain text, under 200 words, nothing else.

Common mistakes when you ask AI for this

  • Getting a 1-10 scale. Asked for a rubric, the AI invents score levels like 'Excellent (5)' that two graders will never agree on. Try: Say: "Binary only: each line is 'Pass if ...; Fail if ...'. No scores or scales."
  • Missing the spec change. The spec still mentions the old 300-word limit, and the AI may grade against it. Try: Add: "Use the current v2 limit of 150 words."
  • Not naming the must-cover rules. The AI writes generic criteria like 'accurate' and skips the two failures users actually hit. Try: List them: "Must cover the 150-word limit, unassigned owners and paused notes."
  • Not saying how many. The AI writes 8 to 12 criteria in sections, too many to grade quickly. Try: Be exact: "Exactly 5 numbered criteria, one line each."
  • Getting markdown back. AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker. Try: Ask for "plain text, no bold, no headings, no tables."
  • Trusting the AI to count words. AI is bad at counting. Ask for a total and it often lands 20% over. Try: Cap each line instead: "each line under 35 words."

What a strong prompt includes

  • Asks for binary pass/fail
  • Says how many criteria
  • Names the must-cover rules
  • Limits each line
  • Asks for plain text

Practice this now ↑

Related exercises