Library › AI Evals › Product managers
Write a 4-line pass/fail rubric for AI search answers
Practice: Setting limits and rules. About 5 minutes.
About this exercise
Your store's search box now shows an AI answer above the results, and you need a rubric before the A/B test. Write exactly 4 numbered criteria, each 'N. Pass if ...; Fail if ...', binary only. They must cover: at most 3 products named, every price matches the catalog, out-of-stock items are labeled 'Out of stock', and no ranking words like 'best' or 'top'. Plain text, under 160 words.
Common mistakes when you ask AI for this
- Getting a rating scale. Asked for a rubric, the AI invents 1-5 scores for 'relevance' and 'helpfulness'. Try: Say: "Binary only: each line is 'Pass if ...; Fail if ...'."
- Not naming the must-haves. The AI writes generic search-quality criteria instead of the four rules your team needs. Try: List them: "at most 3 products, catalog prices, Out of stock label, no ranking words."
- Fuzzy checks. A check like 'not too many products' gets different answers from every grader. Try: Use numbers: "Fail if more than 3 products are named."
- Too many criteria. The AI writes 8 or more criteria, including tone. Try: Be exact: "Exactly 4 numbered criteria, one line each."
What a strong prompt includes
- Asks for binary pass/fail
- Names the must-haves
- Says exactly how many
- Limits each line
- Asks for plain text
Related exercises
- Add dark mode by overriding your existing CSS variables (Coding AI · Software developers)
- Add input checks to your number guessing game (Coding AI · High school students)
- Add labels to a signup form without rewriting the page (Coding AI · High school students)
- Announce the employee assistance program in three steps (Writing AI · HR and recruiting)
- Answer a patient portal message about delayed lab results (Writing AI · Nurses and healthcare workers)