{"id":"eval-pm-rubric-ai-shopping-search-answers","audience":"product-managers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a 4-line pass/fail rubric for AI search answers","free":false,"task":"Your store's search box now shows an AI answer above the results, and you need a rubric before the A/B test. Write exactly 4 numbered criteria, each 'N. Pass if ...; Fail if ...', binary only. They must cover: at most 3 products named, every price matches the catalog, out-of-stock items are labeled 'Out of stock', and no ranking words like 'best' or 'top'. Plain text, under 160 words.","materialLabel":"Sample AI answers and team feedback (attached to your prompt automatically)","material":"Fennick Home - AI search answers, internal dogfood (Nov 3)\n\nQUERY: \"quiet dishwasher under $700\"\nAI ANSWER: \"The best quiet dishwashers under $700 are the Fennick Q40 ($649), the Arden Silent 3 ($599), the Brio 45dB ($689) and the Halden S2 ($679). The Q40 is the top pick for most kitchens.\"\nCatalog check: Q40 is $669, Halden S2 is out of stock.\n\nQUERY: \"blackout curtains for nursery\"\nAI ANSWER: \"Try the Lumen Blackout Panel ($34) or the DreamDark 2-pack ($52). Both block about 99% of light.\"\nCatalog check: prices correct, both in stock.\n\nTeam feedback\n- Merch: \"It ranks things. We have vendor agreements, we can't call one brand 'best'.\"\n- Pricing: \"Wrong prices are a legal problem in two states.\"\n- UX: \"4+ products is just the results list again, pointless.\"\n- Ops: \"It recommended something we can't ship.\"\n- Design: \"Love the friendly tone, keep it.\"\n\nA/B test starts Nov 17 with 5% of traffic. You'll grade 200 answers before then.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":160}],"mistakes":[{"title":"Getting a rating scale","why":"Asked for a rubric, the AI invents 1-5 scores for 'relevance' and 'helpfulness'.","fix":"Say: \"Binary only: each line is 'Pass if ...; Fail if ...'.\"","checks":["r0","r2"]},{"title":"Not naming the must-haves","why":"The AI writes generic search-quality criteria instead of the four rules your team needs.","fix":"List them: \"at most 3 products, catalog prices, Out of stock label, no ranking words.\"","checks":["r1"]},{"title":"Fuzzy checks","why":"A check like 'not too many products' gets different answers from every grader.","fix":"Use numbers: \"Fail if more than 3 products are named.\"","checks":["r0"]},{"title":"Too many criteria","why":"The AI writes 8 or more criteria, including tone.","fix":"Be exact: \"Exactly 4 numbered criteria, one line each.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for binary pass/fail","patterns":["pass if","pass/fail","binary","no (score|scale)"]},{"label":"Names the must-haves","patterns":["3 products|three products","catalog","out of stock","best|ranking"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(criteri|line)","exactly \\d+"]},{"label":"Limits each line","patterns":["\\d+\\s*words","one line"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A rubric should test the failures your team can't live with. Name the four rules, use numbers where you can, and ask for binary lines.","rubricLabels":["Every criterion is binary with a concrete condition checkable by reading one answer next to the catalog (no 1-5 or 1-10 scales, no points, no 'partially', no test that is only a vague word like 'helpful' or 'relevant')","Covers all four required topics: at most 3 products, every price matches the catalog, out-of-stock items labeled 'Out of stock', and no ranking words like 'best' or 'top'","Each of the 4 lines has both parts on the same line in the shape 'Pass if ...; Fail if ...', and no criterion is about tone"]}