{"id":"eval-pm-rubric-meeting-recap-five-binary","audience":"product-managers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a 5-line pass/fail rubric for an AI meeting recap","free":false,"task":"You own the AI Recap feature at Tallow Notes and need a rubric your team can grade recaps with. Write exactly 5 numbered criteria, each one line in the form 'Pass if ...; Fail if ...', binary only (no scores or scales). The criteria must cover the current word limit, action items with no named owner, and the paused part of the meeting. Plain text, under 200 words, nothing else.","materialLabel":"The feature spec and user feedback (attached to your prompt automatically)","material":"AI RECAP - feature spec v2 (owner: you)\nWhat it does: after a call ends, the AI writes a recap that is emailed to every attendee.\n\nContents\n- 1-sentence purpose of the meeting\n- Decisions made\n- Action items, each with an owner and a due date if one was said\n- If nobody was named for an action item, the recap writes \"Owner: unassigned\". It must never guess an owner from who talked most.\n\nLimits\n- v1 limit was 300 words. CHANGED in v2 (Oct 30): max 150 words, because people stopped reading long recaps.\n- Anything said while the host had \"Pause notes\" turned on is off the record and must not appear anywhere in the recap.\n- Recap is written in the language the meeting was held in.\n\nFeedback from beta users (last 2 weeks)\n- \"It said Priya owns the vendor follow-up. Nobody said that. Priya was just talking a lot.\" (Dana, ops)\n- \"The recap mentioned the layoff rumor we discussed with notes paused. Very bad.\" (Marco, HR lead)\n- \"Too long, I skim the first 3 lines.\" (Kenji, sales)\n- \"Love it, saves me 10 min per meeting.\" (Ruth, design)\n- \"Our Spanish standup got an English recap.\" (Lucia, support)\n\nEng notes: average recap today is 212 words. Pause notes ships in 4.18.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":200}],"mistakes":[{"title":"Getting a 1-10 scale","why":"Asked for a rubric, the AI invents score levels like 'Excellent (5)' that two graders will never agree on.","fix":"Say: \"Binary only: each line is 'Pass if ...; Fail if ...'. No scores or scales.\"","checks":["r0","r2"]},{"title":"Missing the spec change","why":"The spec still mentions the old 300-word limit, and the AI may grade against it.","fix":"Add: \"Use the current v2 limit of 150 words.\"","checks":["r1"]},{"title":"Not naming the must-cover rules","why":"The AI writes generic criteria like 'accurate' and skips the two failures users actually hit.","fix":"List them: \"Must cover the 150-word limit, unassigned owners and paused notes.\"","checks":["r1","r0"]},{"title":"Not saying how many","why":"The AI writes 8 to 12 criteria in sections, too many to grade quickly.","fix":"Be exact: \"Exactly 5 numbered criteria, one line each.\"","checks":["numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker.","fix":"Ask for \"plain text, no bold, no headings, no tables.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total and it often lands 20% over.","fix":"Cap each line instead: \"each line under 35 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Asks for binary pass/fail","patterns":["pass if","pass/fail","binary","no (score|scale)"]},{"label":"Says how many criteria","patterns":["\\b(5|five)\\b.{0,25}(criteri|line|item)","exactly \\d+"]},{"label":"Names the must-cover rules","patterns":["150","unassigned","pause"]},{"label":"Limits each line","patterns":["\\d+\\s*words","one line"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A rubric only helps if two people grading the same recap get the same answer. Ask for binary lines, say how many, and name the rules users are actually tripping on.","rubricLabels":["Every criterion is binary pass/fail with a concrete condition someone can check by reading one recap (no 1-10 or 1-5 scales, no points, no 'partially' level, and no criterion whose whole test is a vague word like 'helpful', 'clear' or 'good')","Covers all three required topics: a 150-word limit (not 300), action items with no named owner marked 'unassigned' instead of guessed, and nothing from the paused part of the meeting","Each of the 5 lines has both parts on the same line in the shape 'Pass if ...; Fail if ...' (no separate Pass/Fail sub-bullets, no intro or closing notes)"]}