{"id":"eval-hc-reminder-bot-complaints-to-checks","audience":"healthcare","category":"evaluation","track":"evals","technique":"verification","title":"Turn front-desk complaints about a reminder bot into checks","free":false,"task":"Your clinic's front desk keeps saying the appointment reminder texts 'feel off'. Write exactly 4 numbered checks anyone can apply to one reminder text, each 'N. Fail if [something you can see or count] - e.g. \"quote from the notes\"'. Don't use vague words like off, cold or confusing as the test, and skip anything that isn't about the text itself. Plain text, under 130 words.","materialLabel":"Front-desk notes about the reminder texts (attached to your prompt automatically)","material":"Fernhill Family Clinic - front desk huddle notes, Tue (admin only, no patient details)\n\nDana: the reminder texts feel off. Patients keep calling to ask what they mean.\nLuis: one said \"Your appt w/ Dr. K @ FHC 2mo re: f/u\" - patients don't know what f/u is\nDana: some texts don't say what time the appointment is, just the date\nPriya: one went out in ALL CAPS. \"REMINDER: APPOINTMENT TOMORROW\". Felt like yelling.\nLuis: they're long too, I saw one that was 4 texts in a row\nDana: and there's no way to reschedule in the text, people call instead\nPriya: I do like that it says the doctor's name, keep that\nLuis: also the parking lot is closed next week, someone should tell patients\nDana: the texts feel cold\nPriya: we got 31 calls last week from people confused by texts\n\nContext: the clinic manager wants a checklist to give the vendor by Friday. The vendor updates templates monthly.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":130}],"mistakes":[{"title":"Keeping the vague words","why":"The AI rewrites 'feels off' as 'is clear and warm', which is just as hard to check.","fix":"Say: \"Each check must be something you can see or count in one text.\"","checks":["r0"]},{"title":"No example from the notes","why":"The AI writes generic texting tips instead of tying each check to what staff saw.","fix":"Add: \"Quote an example from the notes on each line.\"","checks":["r1"]},{"title":"Treating every comment as a problem","why":"The AI lists everything, including Priya's praise and the parking news.","fix":"Add: \"Skip praise and anything not about the text itself.\"","checks":["r2"]},{"title":"Too many checks","why":"The AI turns every comment into a check and adds its own.","fix":"Be exact: \"Exactly 4 numbered checks.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Asks for observable checks","patterns":["see or count","observable","countable","testable"]},{"label":"Asks for quoted examples","patterns":["quote","example","e\\.g\\."]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(check|line|criteri)","exactly \\d+"]},{"label":"Skips praise and off-topic","patterns":["praise","doctor'?s name","parking","not about the text"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"'Feels off' can't be checked, but 'uses f/u' can. Ask the AI to turn each complaint into something you can see or count in one text, with a real example.","rubricLabels":["Each check's test is something visible or countable in one text (abbreviations like f/u or appt, a missing time, all-caps words, more than one text segment, no reschedule link or number), not a judgment word like off, cold, confusing or friendly","Each of the 4 lines includes an example quoted from the notes, such as \"f/u\" or \"REMINDER: APPOINTMENT TOMORROW\"","Does not make the doctor's name a failure, and no check is about the parking lot"]}