{"id":"eval-mgr-regression-checklist-model-switch","audience":"managers","category":"evaluation","track":"evals","technique":"context","title":"List what to re-test before switching your team's AI model","free":false,"task":"IT is switching the AI model your team uses to draft client renewal letters, and you want to re-run past problem cases first. Write exactly 4 numbered lines, each 'N. Case-id | Re-run: the saved example | Pass if: ...', using only cases from your log, and include every case marked as having reached a client. Plain text, under 140 words.","materialLabel":"Your team's log of past AI letter problems (attached to your prompt automatically)","material":"Renewals team - AI letter problems log (kept by you since March)\nEach case has a saved example in the shared drive: /renewals/ai-cases/\n\nC-02 Letter used last year's price ($1,200) instead of the renewal price ($1,350). Reached a client: YES. Example: c02_price.docx\nC-05 Letter addressed the client as \"Dear Customer\" instead of the contact's name. Reached a client: no (caught in review). Example: c05_name.docx\nC-07 Letter promised a 10% loyalty discount we don't offer. Reached a client: YES. Example: c07_discount.docx\nC-09 Letter was 2 pages; template limit is 1 page. Reached a client: no. Example: c09_length.docx\nC-11 Letter mentioned another client's company name. Reached a client: YES. Example: c11_wrongclient.docx\nC-14 Letter used British spelling (\"organisation\"). Reached a client: no. Example: c14_spelling.docx\n\nNotes\n- IT says the new model is \"smarter and cheaper\".\n- Your director asked for a quick \"we tested it\" note by Friday.\n- Someone suggested testing the new model on cold sales emails too. Different team.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":140}],"mistakes":[{"title":"Generic test ideas","why":"The AI suggests 'check tone and accuracy' instead of re-running the cases you logged.","fix":"Say: \"Each line names a case id and its saved example.\"","checks":["r1"]},{"title":"Missing the costly cases","why":"The AI picks cases at random and may skip ones that reached clients.","fix":"Add: \"Include every case that reached a client.\"","checks":["r0"]},{"title":"Vague pass conditions","why":"The AI writes 'letter looks right', which tells a reviewer nothing.","fix":"Ask: \"Pass if says the exact result, like 'shows the $1,350 price'.\"","checks":["r2"]},{"title":"A full test plan","why":"The AI writes sections on rollout, training and sign-off.","fix":"Be exact: \"Exactly 4 numbered lines: Case-id | Re-run | Pass if.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Uses the case log","patterns":["C-\\d","case","log","saved"]},{"label":"Prioritizes cases that reached clients","patterns":["reached a client","reached client","yes"]},{"label":"Asks for a pass condition","patterns":["pass if","expected","specific"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|case)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Before a model switch, re-run the failures that already hurt you. Point the AI at your log, say which cases matter most, and ask for a concrete pass condition.","rubricLabels":["Includes C-02, C-07 and C-11 (every case that reached a client) plus one other case from the log","Every line names a case id and its saved example file from the log (for example c07_discount.docx), and there is no line about cold sales emails or anything not in the log","Every Pass if is a specific observable result (for example 'shows the $1,350 renewal price', 'offers no discount', or 'names only this client's company'), not 'letter is correct'"]}