{"id":"eval-col-metric-flashcard-generator-accuracy","audience":"college","category":"evaluation","track":"evals","technique":"context","title":"Pick the right metric for an AI flashcard generator","free":false,"task":"Your study group's AI flashcard generator gets great ratings, but you want to know if the cards are actually right. Your TA says cards must be at least 95% correct against the textbook, and duplicates must stay at or under 5%. Write exactly 4 numbered lines: Metric (not ratings), Threshold (the number and who set it), Result (the math, then Ready or Not ready), and Guardrail (the duplicate math against 5%). Plain text, under 80 words.","materialLabel":"Your audit of the flashcards (attached to your prompt automatically)","material":"BIO 150 study group - AI flashcard generator audit (Ch. 4-6)\n\nCards generated: 120\nChecked against the textbook by Nia and Omar (each card: correct or not)\n- Correct: 102\n- Wrong answer or definition: 18\n\nDuplicates (same question asked twice): 6 of 120\n\nStudent ratings in the app: 4.6 / 5 (35 ratings)\n\nNotes\n- App store page says \"Students love our flashcards!\"\n- Most wrong cards were from Ch. 6 (cell respiration).\n- The generator took 2 minutes; making them by hand took us 3 hours last time.\n- Exam is Nov 20.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Trusting the ratings","why":"The AI sees 4.6 stars and calls the cards great, but ratings don't check facts.","fix":"Say: \"Don't use ratings. Measure correct cards against the textbook.\"","checks":["r0"]},{"title":"Missing the TA's bar","why":"The AI picks its own threshold because it doesn't know your TA's rule.","fix":"Add: \"Threshold: 95% correct, from the TA.\"","checks":["r1"]},{"title":"No duplicate check","why":"The AI doesn't know duplicates matter to you, so it skips them.","fix":"Ask: \"Guardrail: duplicates out of 120 vs the 5% limit.\"","checks":["r2"]},{"title":"A long review","why":"The AI writes study tips and a chapter breakdown.","fix":"Show the shape: \"1. Metric: 2. Threshold: 3. Result: 4. Guardrail:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Rules out ratings","patterns":["not.{0,20}rating","rating","stars"]},{"label":"Measures correctness","patterns":["correct","textbook","accura"]},{"label":"Gives the TA's bar","patterns":["95","ta\\b","threshold"]},{"label":"Adds the duplicate guardrail","patterns":["duplicate","5%","guardrail"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"High ratings don't mean the cards are right. Tell the AI what to measure, give your TA's bar, and ask for the math.","rubricLabels":["The metric is the share of cards that are correct against the textbook, not student ratings","The threshold is 95%, set by the TA, and the Result line shows 102 of 120 = 85% and says Not ready","The Guardrail line shows 6 of 120 = 5% and says it is within (at or under) the 5% limit"]}