Library › AI Evals › Product managers

Plan a weekly human check of AI review summaries

Practice: Setting limits and rules. About 5 minutes.

About this exercise

Your product pages now show AI summaries of customer reviews, and your team can spend 4 hours a week checking them at 3 minutes each. Write exactly 4 numbered lines: Capacity (the math), Always review (the high-risk groups from the data, with their total), Random sample (how many of the rest, drawn across every category), and Sample rate (the math as a percent). Plain text, under 100 words.

Common mistakes when you ask AI for this

  • Checking only the biggest category. The AI accepts last month's habit of checking only Kitchen. Try: Say: "Random sample drawn across every category."
  • Forgetting the must-check summaries. The AI samples at random and misses bestsellers and shopper reports. Try: Add: "Always review the high-risk groups first, then sample the rest."
  • Hiding the math. The AI says 'about 80' or '2%' without showing how. Try: Say: "Show the math on each line, like 4 x 60 / 3 = 80."
  • A long QA process. The AI writes a workflow with roles and dashboards. Try: Be exact: "Exactly 4 numbered lines: Capacity, Always review, Random sample, Sample rate."

What a strong prompt includes

  • Asks for the math
  • Reviews high-risk summaries first
  • Samples every category
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises