{"id":"eval-tch-yes-no-checks-essay-feedback-bot","audience":"teachers","category":"evaluation","track":"evals","technique":"constraints","title":"Write 4 yes/no checks for an AI essay feedback tool","free":false,"task":"Your school is piloting an AI tool that gives 8th graders feedback on essay drafts, and you need to spot-check its feedback, not the essays. Write exactly 4 numbered yes/no questions, each answerable by reading one piece of feedback, that cover the school's rules for the tool. Each line 'N. Yes/No: question?' under 20 words, plain text, nothing else.","materialLabel":"The pilot guidelines and a sample (attached to your prompt automatically)","material":"Ridgeview Middle School - AI Writing Coach pilot, guidelines for teachers\n\nWhy we're piloting: students get feedback between drafts without waiting a week for us.\n\nRules the district set for the tool's feedback\n- It never rewrites the student's sentences for them. It may point to a sentence, but the student does the rewriting.\n- It names one specific strength and quotes the student's own words.\n- It gives no more than 2 next steps, so students aren't overwhelmed.\n- It never gives a grade, score or \"this would get a B\". Grades come from teachers only.\n\nSAMPLE FEEDBACK the tool gave last week (Grade 8, argument essay on school start times)\n\"Great job! Your essay is really strong and well organized. Here's a better version of your opening: 'Every morning, thousands of tired teens drag themselves to school before their brains are awake.' You should also add more evidence, fix your commas, vary your sentence starts, and strengthen your conclusion. Overall this is B+ work!\"\n\nTeacher notes from the first week\n- Mr. Alvarez: students loved the encouragement but copied the rewritten sentences word for word.\n- Ms. Chen: one student asked if the B+ \"counts\".\n- Parent email: \"Is the AI grading my child?\"\n- Pilot check-in is Nov 20.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Asking 'how good is it?'","why":"The AI gives you a 1-5 scale for 'helpfulness' and 'tone' that teachers will score differently.","fix":"Say: \"Each check is a yes/no question you can answer by reading one piece of feedback.\"","checks":["r0"]},{"title":"Not tying checks to the rules","why":"The AI invents general writing criteria like 'addresses organization' instead of your district's 4 rules.","fix":"Add: \"Cover the 4 district rules, one question each.\"","checks":["r1"]},{"title":"Fuzzy checks","why":"A check like 'is it specific?' can't be answered the same way twice.","fix":"Make it countable: \"Does it quote the student's words? Does it give 2 or fewer next steps?\"","checks":["r2"]},{"title":"Getting a long guide","why":"The AI writes an explanation of each rule and tips for teachers.","fix":"Be exact: \"Exactly 4 numbered lines, each 'N. Yes/No: question?', nothing else.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for yes/no questions","patterns":["yes/no","yes or no","binary","pass/fail"]},{"label":"Ties checks to the district rules","patterns":["rule","district","guideline"]},{"label":"Makes checks countable","patterns":["quote","\\b2\\b|two","count"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(question|line|check)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A good spot-check gives every teacher the same answer for the same feedback. Ask for yes/no questions tied to each district rule, using things you can count or quote.","rubricLabels":["Each of the 4 lines is a yes/no question in the shape 'N. Yes/No: ...?' that can be answered by reading one piece of feedback (no 1-5 ratings, no 'how well' or 'to what extent' questions)","Together the 4 questions cover all 4 district rules: no rewritten student sentences, one specific strength that quotes the student, at most 2 next steps, and no grade or score","The strength question asks whether the feedback quotes the student's own words, and the next-steps question uses the number 2"]}