{"id":"evals2-define-spot-the-vibes-check","audience":"small-business","category":"evaluation","track":"evals","stage":"define-success","level":"beginner","technique":"verification","title":"Spot which of 4 proposed evals is really a vibes check","free":false,"task":"Lena Ortiz owns Petal & Post, a florist, and her consultant sent four ways to \"evaluate\" the order chatbot Posy. Lena asks which one is just a vibes check. Write a plain-text Slack reply under 90 words: first the line \"Hi Lena,\", then exactly 4 labeled lines: \"Vibes check: [letter]\", \"Missing: [what it lacks, in one sentence]\", \"Fix: [one sentence under 25 words]\" and \"Keep: [the other three letters]\".","materialLabel":"The consultant's four options (attached to your prompt automatically)","material":"Posy chatbot - proposed evaluation methods (from the consultant's email)\n\nA. Every Friday, three staff each pick 10 chats they like, rate the bot 1 to 5 for \"overall quality\", and we average the scores. The average is the eval number we track.\n\nB. Keep a fixed list of 80 customer questions with the right answer written down from the price list. After every change, run all 80 and count how many the bot gets right.\n\nC. After every change, check that each order reply contains a pickup date in the format MM/DD using a simple text pattern, and report the percent that do.\n\nD. Use a second AI to score each reply against a 4-point checklist (right price, right delivery fee, right date, polite close) on the same fixed 80 questions after every change.\n\nBackground\n- Posy takes bouquet orders and answers delivery-fee questions.\n- Valentine's week is the busiest of the year.\n- Lena's worry: \"Last spring we thought Posy was great, then three wrong delivery fees cost us $140.\"","rules":[{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Calling several options vibes","why":"The AI says A and C are both weak because C is narrow.","fix":"Say: \"Exactly one is a vibes check: the one with no fixed cases and no written answers.\"","checks":["r0"]},{"title":"Vague complaints","why":"The AI says A is \"subjective\" without naming what is missing.","fix":"Add: \"Missing: no fixed cases and no written correct answers.\"","checks":["r1"]},{"title":"Advice with no fix","why":"The AI writes a paragraph about better evaluation in general.","fix":"Add: \"Fix: one sentence under 25 words, scoring the same questions against written answers.\"","checks":["r2"]},{"title":"Chatty Slack message","why":"The AI adds greetings, emoji and offers more help.","fix":"Say: \"Hi Lena, then exactly 4 labeled lines, plain text, under 90 words.\"","checks":["maxWords","plainText"]}],"promptHints":[{"label":"Names the 4 labeled lines","patterns":["vibes check","missing","fix","keep"]},{"label":"Addresses Lena","patterns":["lena|slack"]},{"label":"Asks for one letter","patterns":["one letter|just one|exactly one|which one"]},{"label":"Caps the length","patterns":["under \\d+ words|\\d+ words"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"A vibes check has no fixed cases and no written right answers. Tell the AI the four labeled lines you want and what to say is missing.","mission":"Tell the owner which proposed eval is only vibes.","musts":["Reply starts Hi Lena","Four labeled lines","Name what is missing"],"rubricLabels":["The reply starts with \"Hi Lena,\" and then has a line exactly \"Vibes check: A\" (no other letter)","The Missing line says there are no written correct answers and no fixed set of cases (it may mention staff choosing their own chats or the 1-to-5 impression)","The Fix line is one sentence under 25 words about scoring the same fixed questions against written correct answers, and the Keep line lists B, C and D"]}