{"id":"eval-col-golden-answer-standard-error","audience":"college","category":"evaluation","track":"evals","technique":"grounding","title":"Write a golden reference answer for a stats tutor bot","free":false,"task":"You're a TA building test cases for your Stats 101 tutor bot, and this question needs a golden answer to grade the bot against. TutorBot always ends with a check question, so write exactly 3 numbered lines: '1. Formula: ...', '2. Plug in: ...' with the numbers worked out, and '3. Check: ...' asking what happens to the SE if the sample grows to 100 people, without giving the answer. Use the course notes, including the corrected formula. Plain text, under 60 words.","materialLabel":"The test question and course notes (attached to your prompt automatically)","material":"TEST CASE #14 for TutorBot (Stats 101, Week 6)\nStudent question: \"My sample has 25 people and the standard deviation is 10. What's the standard error of the mean?\"\n\nCOURSE NOTES - Week 6: Standard error (Prof. Lindqvist)\n- The standard error (SE) of the mean tells you how much the sample mean would bounce around if you took many samples of the same size.\n- Formula on slide 4: SE = s / n\n- CORRECTION posted on the course page, Oct 28: slide 4 had a typo. The formula is SE = s / √n (divide by the square root of the sample size).\n- Bigger samples give smaller SE, so the sample mean is a more precise estimate.\n- Notation in this course: s = sample standard deviation, n = sample size.\n\nWhat's next\n- Week 7: confidence intervals (using SE with z = 1.96). Don't introduce this yet; students haven't seen it.\n\nTA notes\n- Bot answers in last week's test run were 150-300 words, way too long for a golden answer.\n- Golden answers are what we compare the bot's answer to, so they should be the shortest complete correct answer.\n- 3 students emailed confused by the slide 4 typo.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":60}],"mistakes":[{"title":"Copying the typo","why":"Slide 4's wrong formula is still in the notes, and the AI may use it and get 0.4.","fix":"Add: \"Use the corrected formula from Oct 28.\"","checks":["r0"]},{"title":"A full lesson instead of a reference","why":"The AI explains standard error at length, which makes a poor answer to grade against.","fix":"Show the shape: \"1. Formula: 2. Plug in: 3. Check:\"","checks":["r1","numberedItems","maxWords"]},{"title":"Forgetting the check question","why":"The AI doesn't know TutorBot ends with a question, so it ends with a summary or gives the answer away.","fix":"Add: \"End with Check: what happens to SE if n is 100? Don't answer it.\"","checks":["r2"]},{"title":"Markdown math","why":"The AI formats formulas with bold or LaTeX that won't show up in your test file.","fix":"Ask for \"plain text, no bold, no LaTeX.\"","checks":["plainText"]}],"promptHints":[{"label":"Uses the correction","patterns":["correct","√|sqrt|square root","typo"]},{"label":"Shows the three-line shape","patterns":["formula","plug in","check"]},{"label":"Asks for the check question","patterns":["check","\\b100\\b","question"]},{"label":"Keeps it short","patterns":["\\d+\\s*words","short","one sentence"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer is the shortest complete correct answer you'll grade the bot against. Show its exact shape, point to the corrected formula, and keep it inside this week's material.","rubricLabels":["Uses the corrected formula SE = s / √n (not s / n) and gets SE = 10 / √25 = 10 / 5 = 2","Has exactly the three lines labeled Formula, Plug in and Check, and does not mention confidence intervals, z-scores or 1.96","The Check line asks what happens to the standard error if the sample grows to 100 people, and does not give the answer"]}