{"id":"eval-dev-regression-checklist-rag-chunk-size","audience":"developers","category":"evaluation","track":"evals","technique":"context","title":"Write a regression checklist before shrinking RAG chunks","free":false,"task":"You're about to cut your docs bot's retrieval chunks from 1,000 to 300 tokens. Write a regression checklist of exactly 4 numbered lines, each 'N. Q-id | Re-run: the saved question | Pass if: ...', covering the saved questions most likely to break when answers get split across chunks. Use only questions from the eval set. Plain text, under 150 words.","materialLabel":"The change and the saved eval questions (attached to your prompt automatically)","material":"docsbot - retrieval change proposal (PR #611)\nChange: chunk_size 1000 -> 300 tokens, overlap 0 -> 50. Goal: cheaper, more precise retrieval for short lookups.\n\nSAVED EVAL QUESTIONS (evals/docsbot_golden.jsonl)\nQ-04 \"What's the default timeout for the API client?\" Answer is one line in config.md.\nQ-09 \"List all 7 steps to rotate an API key.\" Steps span about 600 tokens in security.md.\nQ-11 \"What does error code E212 mean?\" One line in errors.md.\nQ-15 \"Compare the Free, Team and Enterprise plan limits.\" Answer is a 40-row table in pricing.md, about 900 tokens.\nQ-18 \"How do I install the CLI on Windows?\" Short section, about 150 tokens.\nQ-22 \"What changed between API v2 and v3?\" Migration guide, 5 sections, about 1,200 tokens total.\nQ-27 \"Is SSO available on the Team plan?\" One line inside the pricing table in pricing.md.\nQ-31 \"What's the rate limit for the search endpoint?\" One line in limits.md.\n\nTeam notes\n- Mo: \"Short lookups like Q-04 and Q-11 should get better.\"\n- Last chunk-size change (2000 -> 1000) broke the plan table answer for a week.\n- Ines asked to also test the new Go SDK docs. They aren't in the eval set yet.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":150}],"mistakes":[{"title":"Generic test ideas","why":"The AI suggests 'test retrieval quality' instead of re-running your saved questions.","fix":"Say: \"Each line names a question id from the eval set.\"","checks":["r1"]},{"title":"Testing the wrong questions","why":"The AI picks the short lookups, which are the ones the change should help, not break.","fix":"Add: \"Pick the questions whose answers are long or split across a big section.\"","checks":["r0"]},{"title":"Vague pass conditions","why":"The AI writes 'answer is accurate', which tells a reviewer nothing.","fix":"Ask: \"Pass if says the exact result, like 'lists all 7 steps in order'.\"","checks":["r2"]},{"title":"A full test plan","why":"The AI writes sections on load testing and monitoring.","fix":"Be exact: \"Exactly 4 numbered lines: Q-id | Re-run | Pass if.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Uses the saved questions","patterns":["Q-\\d","saved","eval set","golden"]},{"label":"Targets what the change risks","patterns":["long","split","span","table"]},{"label":"Asks for a pass condition","patterns":["pass if","expected","specific"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|item|question)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Smaller chunks help short lookups and can break long answers. Point the AI at your saved questions, say which kind of answer is at risk, and ask for a concrete pass condition.","rubricLabels":["The 4 lines are Q-09, Q-15, Q-22 and Q-27 (the answers that are long or sit inside the long pricing table), in any order","Every line names a question id and its saved question from the eval set, and there is no line about the Go SDK or any question not in the set","Every Pass if is a specific observable result (for example 'lists all 7 steps in order', 'includes all three plans' limits', or 'says whether Team has SSO correctly per the table'), not 'answers correctly'"]}