Library › AI Evals › Software developers

Write a regression checklist before shrinking RAG chunks

Practice: Giving the right context. About 5 minutes.

About this exercise

You're about to cut your docs bot's retrieval chunks from 1,000 to 300 tokens. Write a regression checklist of exactly 4 numbered lines, each 'N. Q-id | Re-run: the saved question | Pass if: ...', covering the saved questions most likely to break when answers get split across chunks. Use only questions from the eval set. Plain text, under 150 words.

Common mistakes when you ask AI for this

  • Generic test ideas. The AI suggests 'test retrieval quality' instead of re-running your saved questions. Try: Say: "Each line names a question id from the eval set."
  • Testing the wrong questions. The AI picks the short lookups, which are the ones the change should help, not break. Try: Add: "Pick the questions whose answers are long or split across a big section."
  • Vague pass conditions. The AI writes 'answer is accurate', which tells a reviewer nothing. Try: Ask: "Pass if says the exact result, like 'lists all 7 steps in order'."
  • A full test plan. The AI writes sections on load testing and monitoring. Try: Be exact: "Exactly 4 numbered lines: Q-id | Re-run | Pass if."

What a strong prompt includes

  • Uses the saved questions
  • Targets what the change risks
  • Asks for a pass condition
  • Says exactly how many
  • Asks for plain text

Practice this now ↑

Related exercises