{"id":"eval-pm-vague-digest-feedback-to-checks","audience":"product-managers","category":"evaluation","track":"evals","technique":"verification","title":"Turn \"the digest feels generic\" into 4 testable checks","free":false,"task":"Users say your AI weekly digest 'feels generic', and you need checks a grader can apply to one digest. Write exactly 4 numbered checks, each 'N. Fail if [something you can see or count in one digest] - e.g. \"quote from the interviews\"'. Don't use vague words like generic, bland or boring as the test, and skip praise and requests for new features. Plain text, under 130 words.","materialLabel":"User interview snippets about the digest (attached to your prompt automatically)","material":"Loomwell (project tool) - AI Weekly Digest, user interview snippets (Oct)\n\nP1 (eng manager): \"It feels generic. It said 'The team made great progress this week' - which team? what progress?\"\nP2 (designer): \"It never names a single project. Just 'several initiatives moved forward'.\"\nP3 (ops lead): \"Every week it opens with 'Here's your weekly summary!' I skip the first line now.\"\nP4 (PM): \"It doesn't say what changed since last week, so I can't tell if anything moved.\"\nP5 (founder): \"I like that it's short, keep it short.\"\nP6 (eng): \"It says 'some tasks are at risk' but doesn't say which ones or who owns them.\"\nP7 (designer): \"Could it also make a slide deck? That'd be cool.\"\nP8 (PM): \"It feels bland. Same words every week: 'great progress', 'moving forward', 'on track'.\"\n\nContext: the digest goes to 4,000 workspace admins every Monday. You want a grader to check 100 digests before the next model update.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":130}],"mistakes":[{"title":"Keeping the vague words","why":"The AI rewrites 'generic' as 'specific and engaging', which is just as hard to grade.","fix":"Say: \"Each check must be something you can see or count in one digest.\"","checks":["r0"]},{"title":"No example from real data","why":"The AI writes generic writing tips instead of tying checks to what users said.","fix":"Add: \"Quote an example from the interviews on each line.\"","checks":["r1"]},{"title":"Treating every comment as a problem","why":"The AI turns P5's praise and P7's feature idea into checks.","fix":"Add: \"Skip praise and feature requests.\"","checks":["r2"]},{"title":"Too many checks","why":"The AI writes one check per interview, plus some of its own.","fix":"Be exact: \"Exactly 4 numbered checks.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for observable checks","patterns":["see or count","observable","countable","testable"]},{"label":"Asks for quoted examples","patterns":["quote","example","e\\.g\\."]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(check|line|criteri)","exactly \\d+"]},{"label":"Skips praise and feature ideas","patterns":["praise","feature","slide","short"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"'Generic' can't be graded, but 'names no project' can. Ask the AI to turn each complaint into something you can see or count in one digest, with a real quote.","rubricLabels":["Each check's test is something visible or countable in one digest (no project names, a stock opener like \"Here's your weekly summary!\", no change vs last week, an at-risk item with no task or owner named, stock phrases like \"great progress\"), not a judgment word like generic, bland, boring or engaging","Each of the 4 lines includes an example quoted from the interviews, such as \"several initiatives moved forward\" or \"some tasks are at risk\"","No check penalizes the digest for being short, and no check is about slide decks or other new features"]}