Library › AI Evals › Managers and team leads

Turn "the bot sounds weird" into 4 testable checks

Practice: Catching mistakes and made-up facts. About 5 minutes.

About this exercise

Your team keeps saying the internal AskOps policy bot 'sounds weird', and you want checks anyone can apply to a single bot answer. Write exactly 4 numbered criteria, each 'N. Fail if [something you can see or count in one answer] - e.g. "quote from the thread"'. Don't use vague words like natural, robotic or weird as the test, and don't turn praise into a failure. Plain text, under 130 words.

Common mistakes when you ask AI for this

  • Keeping the vague words. The AI rewrites 'robotic' as 'sounds natural and human', which is just as hard to test. Try: Say: "Each criterion must be something you can see or count in one answer."
  • No example from real data. The AI writes generic criteria instead of tying each one to what people actually saw. Try: Add: "Quote an example from the thread on each line."
  • Treating every comment as a problem. The AI lists everything in the thread, including Omar's praise and the vendor question. Try: Add: "Skip praise and the vendor question."
  • Not saying how many. The AI turns every message into its own criterion, plus a few it invents. Try: Be exact: "Exactly 4 numbered criteria."
  • Getting markdown back. AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker. Try: Ask for "plain text, no bold, no headings, no tables."
  • Trusting the AI to count words. AI is bad at counting. Ask for a total and it often lands 20% over. Try: Cap each line instead: "each line under 20 words."

What a strong prompt includes

  • Asks for observable checks
  • Asks for quoted examples
  • Says how many
  • Skips praise and off-topic
  • Asks for plain text

Practice this now ↑

Related exercises