{"id":"eval-tch-error-analysis-ai-quiz-questions","audience":"teachers","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 flagged AI quiz questions into 3 failure types","free":false,"task":"You flagged 13 questions from an AI quiz generator and wrote a note on each. Group them into exactly 3 numbered failure types, each line 'Type name: #ids = count', then one last line 'Total: N'. Leave out any note you later took back. Plain text, under 70 words.","materialLabel":"Your notes on the flagged questions (attached to your prompt automatically)","material":"Unit 4 quiz (Ecosystems, grade 6) - AI generator test, Mr. Basu\nMy notes on each flagged question:\n\n#1 answer key says B, but the right answer is C (decomposers)\n#2 asks about photosynthesis equations - we don't cover that until Unit 6\n#3 both A and D could be right (\"producer\" and \"plant\")\n#4 key marks \"omnivore\" wrong for bears, it's right\n#5 question on DNA - not in this unit at all\n#6 \"Which is the most important animal?\" - no single right answer\n#7 answer key says food chains start with consumers, wrong\n#8 took this one back, it's fine, I misread the diagram\n#9 asks about plate tectonics, that's earth science, not our unit\n#10 two choices both describe a herbivore\n#11 key says 4 trophic levels, the diagram shows 3\n#12 asks students to compare biomes, that's Unit 5\n#13 \"Which best describes a food web?\" - B and C are nearly identical\n\nNotes: the quiz is 25 questions. I want to tell the vendor which problem is worst. Department meeting Tuesday.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI makes a separate group for almost every note, so nothing stands out to report.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes the problems but you can't tell which questions belong where.","fix":"Show the shape: \"Type name: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting the note you took back","why":"The AI counts every numbered note, including #8.","fix":"Add: \"Leave out notes I took back, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold type names and headings.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Handles the withdrawn note","patterns":["took back","#8","leave out","exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"To tell a vendor what's worst, you need a short list of failure types with counts. Fix the number of types, ask for the ids behind each count, and drop the note you took back.","rubricLabels":["Uses these 3 groups (names may differ): wrong answer key #1, #4, #7, #11 = 4; off-unit topic #2, #5, #9, #12 = 4; more than one right answer or no right answer #3, #6, #10, #13 = 4","Leaves out #8 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}