{"id":"eval-col-error-analysis-ai-research-assistant","audience":"college","category":"evaluation","track":"evals","technique":"format","title":"Sort 13 bad AI research answers into 3 failure types","free":false,"task":"For your info-science class, you logged 13 bad answers from an AI research assistant and noted what went wrong. Group them into exactly 3 numbered failure types, each line 'Type: #ids = count', then one last line 'Total: N'. Leave out any note you later found was your own mistake. Plain text, under 70 words.","materialLabel":"Your log of bad answers (attached to your prompt automatically)","material":"INFO 220 project - AI research assistant failure log (Kenji + Ava)\n\n#1 cited a journal article that doesn't exist (no DOI, not in the library)\n#2 said a 2019 study found X, but the abstract says the opposite\n#3 answered about the UK when the question asked about Canada\n#4 gave a fake book title by a real author\n#5 said the study had 5,000 participants, paper says 500\n#6 my mistake: I typed the wrong question, the answer was fine for what I typed\n#7 question was about teens, answer was about adults\n#8 invented a URL that 404s\n#9 said the finding was \"significant\", paper says not significant\n#10 question asked for 2020 onward, answer used 2008 sources only\n#11 listed a conference paper that the conference site doesn't have\n#12 summary says the authors recommend banning phones; the paper makes no recommendation\n#13 asked about public libraries, answered about university libraries\n\nContext: our final report has to name the most common failure and suggest a fix.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Too many types","why":"The AI splits fake books, fake URLs and fake papers into separate groups.","fix":"Say: \"Exactly 3 failure types.\"","checks":["numberedItems","r0"]},{"title":"No ids or counts","why":"The AI describes patterns but not which answers belong where.","fix":"Show the shape: \"Type: #1, #4 = 2\".","checks":["r2"]},{"title":"Counting your own mistake","why":"The AI counts every note, including the one where you typed the wrong question.","fix":"Add: \"Leave out notes that were my mistake, and end with Total: N.\"","checks":["r1"]},{"title":"Getting markdown back","why":"AI defaults to bold headings and long bullets.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Says exactly how many","patterns":["\\b(3|three)\\b.{0,25}(type|group|categor)","exactly \\d+"]},{"label":"Asks for ids and counts","patterns":["#ids","\\bids\\b","count","how many"]},{"label":"Leaves out your mistake","patterns":["#6","my (own )?mistake","leave out|exclude"]},{"label":"Asks for a total","patterns":["total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Error analysis turns a log into a short list of failure types with counts. Fix the number of types, ask for the ids behind each count, and drop the note that wasn't the AI's fault.","rubricLabels":["Uses these 3 groups (names may differ): made-up source #1, #4, #8, #11 = 4; misstated what a real source says #2, #5, #9, #12 = 4; answered a different question than asked (place, group, years or type) #3, #7, #10, #13 = 4","Leaves out #6 and ends with a line saying Total: 12","Each type line lists its # ids followed by the count, and each count equals the number of ids on that line"]}