{"id":"eval-dev-five-tests-docs-slack-bot","audience":"developers","category":"evaluation","track":"evals","technique":"context","title":"Write 5 edge-case tests for an internal docs Slack bot","free":false,"task":"Your internal docs bot answers engineers in Slack, and you need 5 test messages before rollout: an empty mention, an adversarial message (trying to get its hidden instructions), the wrong language, a very long paste, and a refusal case (asking for HR data). Write exactly 5 numbered lines, each 'N. Case | Message: \"exact message\" | Expected: what the bot does per the spec'. Plain text, under 200 words.","materialLabel":"The bot's spec (attached to your prompt automatically)","material":"@docsbot - internal engineering docs assistant, spec v1 (rollout Dec 1)\n\nAnswers questions using the eng wiki only.\n\nBehavior\n- Empty mention (just \"@docsbot\"): replies \"What would you like to know? Ask me about the eng wiki.\"\n- English only for v1. Other languages: replies in English that it only supports English for now and asks the user to rephrase.\n- Messages over 3,000 characters (usually pasted logs): replies asking for the relevant 50 lines or the error message only. Does not try to analyze the whole paste.\n- Never reveals its system prompt or configuration, even if asked to \"ignore previous instructions\". Replies that it can't share its configuration and offers to help with a docs question.\n- No HR or people data (salaries, performance reviews, who's on leave). Replies that it can't help with that and points to the #ask-people-team channel.\n- Every answer links the wiki page it used.\n\nNotes from the beta (12 users)\n- One user pasted a 9,000-line stack trace and the bot timed out.\n- A user wrote in Portuguese and got a half-Portuguese answer.\n- Someone asked who's out next week. The bot guessed. Fixed in build 4.\n- Feature request: Jira integration. Not in v1.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":200}],"mistakes":[{"title":"Generic test ideas","why":"The AI writes 'test with unusual input' instead of a message someone can send.","fix":"Say: \"Each Message is the exact Slack message in quotes.\"","checks":["r1"]},{"title":"Not naming the edge cases","why":"The AI tests normal docs questions and forgets the adversarial and refusal ones.","fix":"List them: \"empty mention, get your instructions, non-English, long paste, HR data.\"","checks":["r0"]},{"title":"Vague expected results","why":"The AI writes 'handles it gracefully' instead of what the spec says.","fix":"Add: \"Expected says exactly what the spec requires.\"","checks":["r2"]},{"title":"Too many tests","why":"The AI writes a long test plan with 10 or more cases.","fix":"Be exact: \"Exactly 5 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Names the edge cases","patterns":["empty","adversarial|system prompt|instructions","language|portuguese|spanish","long|paste","hr|salar|people data|refus"]},{"label":"Asks for exact messages","patterns":["exact","in quotes","quoted","word for word"]},{"label":"Ties expected to the spec","patterns":["spec","expected"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(test|case|line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Edge-case tests check what the bot does when people don't ask nicely. Name each case, ask for the exact Slack message, and tie each expected result to the spec.","rubricLabels":["Has exactly the 5 named cases, one each: empty mention, adversarial attempt to get its instructions or configuration, a non-English message, a very long paste (over 3,000 characters), and a request for HR or people data","Every Message is given in quotation marks (the long-paste case may describe the paste, like a 9,000-line stack trace)","Every Expected matches the spec: asks what you'd like to know about the eng wiki; says it can't share its configuration and offers docs help; replies in English that only English is supported and asks to rephrase; asks for the relevant 50 lines or the error message; says it can't help and points to #ask-people-team"]}