{"id":"eval-mgr-vague-bot-feedback-to-criteria","audience":"managers","category":"evaluation","track":"evals","technique":"verification","title":"Turn \"the bot sounds weird\" into 4 testable checks","free":false,"task":"Your team keeps saying the internal AskOps policy bot 'sounds weird', and you want checks anyone can apply to a single bot answer. Write exactly 4 numbered criteria, each 'N. Fail if [something you can see or count in one answer] - e.g. \"quote from the thread\"'. Don't use vague words like natural, robotic or weird as the test, and don't turn praise into a failure. Plain text, under 130 words.","materialLabel":"Your team's Slack thread about the bot (attached to your prompt automatically)","material":"#ops-team thread, Tues\n\nMaya: honestly AskOps sounds weird lately\nBen: yeah it's so robotic. it opened with \"Dear valued employee,\" when I asked about PTO lol\nMaya: and it keeps saying \"As an AI language model, I cannot...\" before it answers anyway\nInes: my problem is the walls of text. I asked if Friday is a holiday and got like 6 paragraphs\nBen: also the exclamation marks!!! \"Great question!!! Happy to help!!!\"\nOmar: one thing I DO like is that it links the actual policy page at the bottom, keep that\nInes: it also told me \"per section 4.2.1(b)\" which means nothing to me, just tell me the answer\nMaya: should we switch vendors? I heard Northwind's bot is better\nOmar: idk, I'd rather fix this one first\nBen: same. also it ended one answer with \"Warm regards, AskOps\" like it's an email\nInes: 6 paragraphs, I counted\n\nContext: AskOps answers HR and IT policy questions for 140 staff. You have 30 minutes on Thursday to agree what \"good\" looks like before IT tweaks the prompt.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":130}],"mistakes":[{"title":"Keeping the vague words","why":"The AI rewrites 'robotic' as 'sounds natural and human', which is just as hard to test.","fix":"Say: \"Each criterion must be something you can see or count in one answer.\"","checks":["r0"]},{"title":"No example from real data","why":"The AI writes generic criteria instead of tying each one to what people actually saw.","fix":"Add: \"Quote an example from the thread on each line.\"","checks":["r1"]},{"title":"Treating every comment as a problem","why":"The AI lists everything in the thread, including Omar's praise and the vendor question.","fix":"Add: \"Skip praise and the vendor question.\"","checks":["r2"]},{"title":"Not saying how many","why":"The AI turns every message into its own criterion, plus a few it invents.","fix":"Be exact: \"Exactly 4 numbered criteria.\"","checks":["numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold labels, headings and tables, which break when you paste into a doc or tracker.","fix":"Ask for \"plain text, no bold, no headings, no tables.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total and it often lands 20% over.","fix":"Cap each line instead: \"each line under 20 words.\"","checks":["maxWords"]}],"promptHints":[{"label":"Asks for observable checks","patterns":["see or count","observable","countable","testable","check(able)? (by|in) (reading|one)"]},{"label":"Asks for quoted examples","patterns":["quote","example","e\\.g\\."]},{"label":"Says how many","patterns":["\\b(4|four)\\b.{0,20}(criteri|line|check)","exactly \\d+"]},{"label":"Skips praise and off-topic","patterns":["praise","policy (page|link)","vendor","positive"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"'Weird' can't be tested, but 'opens with Dear valued employee' can. Ask the AI to turn each complaint into something you can see or count, with a real example.","rubricLabels":["Each criterion's test is something visible or countable in one answer (a specific phrase, a number of paragraphs or exclamation marks, an email-style greeting or sign-off), not a judgment word like natural, robotic, weird, friendly or professional","Each of the 4 lines includes an example quoted from the thread, such as \"Dear valued employee\", \"As an AI language model\" or \"Great question!!!\"","Does not make linking the policy page a failure, and no criterion is about switching vendors"]}