{"id":"study2-chatbot-summary-vs-abstract-labels","audience":"college","category":"fact-checking","technique":"verification","track":"study","level":"advanced","title":"Label each sentence of a chatbot summary against the abstract","free":false,"task":"A chatbot summarized a study for your paper. Compare each of its 4 sentences with the abstract. Get exactly 4 numbered lines, each labeled Faithful, Overstated or Invented followed by a reason under 12 words, judged only by the abstract. Plain text, under 90 words.","materialLabel":"The abstract and the chatbot summary (attached to your prompt automatically)","material":"ABSTRACT (Lopez & Ng, 2023)\nWe compared spaced practice with massed practice in 120 ninth-grade students at one school. Students studied vocabulary over two weeks, either in four short sessions (spaced) or one long session (massed). On a test two weeks later, the spaced group scored 15% higher. On a test immediately after studying, the groups did not differ. Students were assigned by class period, not randomly. We suggest spaced practice may help long-term retention.\n\nCHATBOT SUMMARY\n1. The study compared spaced and massed practice with 120 ninth graders.\n2. It proved that spaced practice works for every student in every subject.\n3. The experiment was run in five countries over two years.\n4. The spaced group did better on the delayed test but not on the immediate one.\n\nI am citing this in my paper on Friday.","rules":[{"type":"numberedItems","count":4},{"type":"maxWords","max":90},{"type":"plainText"}],"mistakes":[{"title":"Trusting the summary","why":"The chatbot sounds confident, so the AI agrees with it.","fix":"Say: \"Judge each sentence only by the abstract.\"","checks":["r1"]},{"title":"Mixing up overstated and invented","why":"Overstated stretches a real finding; invented adds something not in the abstract.","fix":"Add: \"Overstated = stretches a real finding. Invented = not in the abstract.\"","checks":["r1"]},{"title":"No label rules","why":"Without exact labels you get paragraphs of commentary.","fix":"Say: \"Each line starts with Faithful, Overstated or Invented.\"","checks":["r0"]},{"title":"Long reasons","why":"The AI explains each at length.","fix":"Say: \"Reason under 12 words.\"","checks":["r2"]},{"title":"Not saying how many lines","why":"The AI picks its own number, and it usually picks more than you need.","fix":"Be exact: \"Exactly 4 numbered lines.\"","checks":["numberedItems"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total and it often lands 20% over.","fix":"Cap each part: \"each line under 12 words.\"","checks":["maxWords"]},{"title":"Forgetting where it's going","why":"AI formats for a screen, with bold and headings that show up as stray symbols.","fix":"Say: \"Plain text, no bold, no headings.\"","checks":["plainText"]}],"promptHints":[{"label":"Says how many lines","patterns":["\\b(4|four)\\b.{0,15}line"]},{"label":"Gives the three labels","patterns":["faithful.{0,25}overstated.{0,25}invented"]},{"label":"Defines the labels","patterns":["overstated\\s*=","stretch"]},{"label":"Uses the abstract only","patterns":["only.{0,25}abstract"]},{"label":"Limits the reasons","patterns":["under \\d+","\\d+\\s*words"]}],"hint":"Fact-checking a summary needs clear label definitions. Define overstated and invented, and make the abstract the only judge.","mission":"Check whether a chatbot's study summary matches the abstract.","musts":["Four numbered lines","Exact labels","Abstract is the judge"],"rubricLabels":["Has exactly 4 numbered lines, each starting with exactly one label: Faithful, Overstated or Invented","The labels are: sentence 1 Faithful, sentence 2 Overstated, sentence 3 Invented, sentence 4 Faithful","Each reason is under 12 words, refers to the abstract only, and no outside facts about the study are used"]}