{"id":"eval-col-bug-report-ai-tutor-derivative","audience":"college","category":"evaluation","track":"evals","technique":"format","title":"Report a wrong calculus answer from the campus AI tutor","free":false,"task":"The campus AI tutor gave you a wrong derivative, and the math department wants a bug report through its form. Write exactly 5 numbered lines: Question (exact), Tutor's answer (quote), Correct answer (with the rule used), Severity, and Steps to reproduce. Use the department's scale: High = wrong final answer, Medium = right answer with a wrong step, Low = typo or formatting. Plain text, under 110 words.","materialLabel":"Your chat with the tutor and your notes (attached to your prompt automatically)","material":"CALC I AI Tutor - chat log (Nov 4, 10:12pm, session id S-55821)\n\nMe: \"What's the derivative of x^2 · sin(x)?\"\nTutor: \"Using the chain rule, the derivative is 2x · cos(x).\"\nMe: \"Are you sure?\"\nTutor: \"Yes! The derivative of x^2 is 2x and the derivative of sin(x) is cos(x), so multiply them: 2x · cos(x).\"\n\nMy notes\n- Textbook section 3.3, product rule: (fg)' = f'g + fg'.\n- So it should be 2x · sin(x) + x^2 · cos(x).\n- My roommate asked it the same thing on her laptop and got the same wrong answer.\n- I'm pretty sure the tutor is \"just bad at math\" (my friend's opinion).\n- The tutor helped me fine with limits last week.\n\nDepartment form note: \"Please include the exact question so we can reproduce it.\"","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Guessing the severity","why":"The AI picks 'critical' or 'minor' on its own scale.","fix":"Give the scale: \"High = wrong final answer.\"","checks":["r1"]},{"title":"No rule in the correction","why":"The AI gives the right answer but not why, so the department can't see what the tutor misused.","fix":"Ask: \"Name the rule, the product rule, in the correct answer.\"","checks":["r0"]},{"title":"Paraphrasing the chat","why":"The AI summarizes, but the department needs the exact question to reproduce it.","fix":"Say: \"Quote the question and the tutor's answer exactly.\"","checks":["r2"]},{"title":"A long complaint","why":"The AI writes a paragraph about AI tutors being unreliable.","fix":"Be exact: \"Exactly 5 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Lists the fields","patterns":["question","correct answer","severity","reproduce"]},{"label":"Gives the severity scale","patterns":["high","wrong final answer","scale"]},{"label":"Asks for exact quotes","patterns":["quote","exact","word for word"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good bug report lets someone else see the exact failure and trigger it again. Name the fields, give the department's scale, and ask for exact quotes.","rubricLabels":["The Correct answer line gives 2x · sin(x) + x^2 · cos(x) (or the same in other notation) and names the product rule","Severity is High because the final answer was wrong","The Question and Tutor's answer lines quote the chat exactly (\"What's the derivative of x^2 · sin(x)?\" and \"2x · cos(x)\"), and Steps to reproduce say to ask that exact question (session S-55821 or a second device is a plus)"]}