{"id":"eval-hc-grade-appointment-bot-replies","audience":"healthcare","category":"evaluation","track":"evals","technique":"format","title":"Grade 4 scheduling-bot replies after a clinic move","free":false,"task":"You're the front-office lead checking your clinic's scheduling bot. The Elm St clinic moved to 40 Birch Ave on Nov 3, so 'Elm St' now fails C1. Grade the 4 replies in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C1 - \"exact words from the reply\"', naming the first criterion it fails. This is admin only: judge logistics, not medical content. Plain text, under 90 words.","materialLabel":"The QA criteria and 4 bot replies (attached to your prompt automatically)","material":"Fernhill Family Clinic - scheduling bot QA (admin only), Nov 4\n\nCRITERIA\nC1: Gives the correct clinic address.\nC2: Tells the patient to arrive 15 minutes early.\nC3: Gives the reschedule line (555-0190).\n\nLocations (from the Sept handout): Main clinic, 12 Oak Rd. North clinic, 88 Elm St.\n\nREPLY 1 (North clinic, Thu 9am)\n\"You're booked Thursday at 9am at our North clinic, 40 Birch Ave. Please arrive 15 minutes early. To reschedule, call 555-0190.\"\n\nREPLY 2 (North clinic, Fri 2pm)\n\"You're set for Friday at 2pm at 88 Elm St. Please arrive 15 minutes early. Need to change it? Call 555-0190.\"\n\nREPLY 3 (Main clinic, Mon 10am)\n\"See you Monday at 10am at 12 Oak Rd. Please arrive 15 minutes early.\"\n\nREPLY 4 (Main clinic, Tue 3pm)\n\"You're booked Tuesday at 3pm at 12 Oak Rd. To reschedule, call 555-0190.\"\n\nNote from the office manager: one line per reply for the QA log.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Using the old address","why":"The handout still lists Elm St, so the AI fails Reply 1 and passes Reply 2.","fix":"Tell it: \"Since Nov 3, North clinic is 40 Birch Ave. Elm St fails C1.\"","checks":["r0"]},{"title":"No evidence behind a FAIL","why":"The AI writes 'missing info' without the words that show it.","fix":"Say: \"Quote the exact words from the reply that fail it.\"","checks":["r1"]},{"title":"Rewrites instead of grades","why":"The AI rewrites the replies instead of grading them.","fix":"Show the shape: \"2. FAIL - C1 - \\\"quote\\\"\", no rewrites.","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold verdicts and a table.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Shows the line shape","patterns":["FAIL - C","PASS","one line per"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Gives the address change","patterns":["birch","elm st","moved"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|repl)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"When a location changes, the AI grades by the old handout unless you say so. Give the change, show the one-line shape, and ask for the quote behind each fail.","rubricLabels":["Grades Reply 1 PASS, Reply 2 FAIL on C1 (Elm St is the old address), Reply 3 FAIL on C3 (no reschedule line), and Reply 4 FAIL on C2 (no 15-minutes-early instruction)","Every FAIL line quotes exact words from that reply in quotation marks (for example \"88 Elm St\", \"Please arrive 15 minutes early.\" or \"To reschedule, call 555-0190.\")","Each line uses only the shape 'N. PASS' or 'N. FAIL - Cx - \"quote\"' with no extra commentary, rewrites or medical remarks"]}