{"id":"eval-pm-grade-ai-onboarding-emails","audience":"product-managers","category":"evaluation","track":"evals","technique":"format","title":"Grade 4 AI onboarding emails against 3 criteria","free":false,"task":"Your AI writes welcome emails for new Tasklane signups, and you're grading this week's sample. On Nov 1 the Starter plan was renamed Basic, so 'Starter' now fails C1. Grade the 4 emails in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C2 - \"exact words from the email\"', naming the first criterion it fails. Plain text, under 90 words.","materialLabel":"The criteria and 4 AI emails (attached to your prompt automatically)","material":"Tasklane - AI welcome email QA (week of Nov 3)\n\nCRITERIA\nC1: Uses the user's current plan name correctly.\nC2: Includes exactly one call to action (one link or button to click).\nC3: Under 80 words.\n\nEMAIL 1 (user: Noor, plan: Basic)\n\"Welcome to Tasklane, Noor! Your Basic plan is ready. Create your first board in under a minute: tasklane.example/start. We're glad you're here.\"\n\nEMAIL 2 (user: Felix, plan: Basic)\n\"Hi Felix, welcome aboard! Your Starter plan includes 3 boards and unlimited tasks. Get going here: tasklane.example/start.\"\n\nEMAIL 3 (user: Ama, plan: Pro)\n\"Welcome, Ama! Your Pro plan is live. Create a board: tasklane.example/start. Invite your team: tasklane.example/invite. Watch the tour: tasklane.example/tour.\"\n\nEMAIL 4 (user: Gus, plan: Pro)\n\"Hi Gus, welcome to Tasklane Pro! Start your first board here: tasklane.example/start. Questions? Just reply to this email.\"\n\nNotes\n- Last month's grading was done as paragraphs and nobody could compare weeks.\n- The growth team wants to test emoji next month.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Missing the rename","why":"The AI doesn't know Starter became Basic, so it passes Email 2.","fix":"Tell it: \"Since Nov 1, Starter is called Basic. Starter fails C1.\"","checks":["r0"]},{"title":"No evidence behind a FAIL","why":"The AI writes 'fails C2' without showing why, so nobody can check the grade.","fix":"Say: \"Quote the exact words from the email that fail it.\"","checks":["r1"]},{"title":"Paragraphs instead of lines","why":"The AI writes a review of each email with suggestions.","fix":"Show the shape: \"2. FAIL - C1 - \\\"quote\\\"\", one line per email.","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to bold verdicts and a table.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Shows the line shape","patterns":["FAIL - C","PASS","one line per"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Gives the plan rename","patterns":["starter","basic","renamed"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|email)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Grades are only comparable week to week if they use the same short shape. Show the line, ask for the quote behind each fail, and tell the AI what changed.","rubricLabels":["Grades Email 1 PASS, Email 2 FAIL on C1 (\"Starter\" is the old plan name), Email 3 FAIL on C2 (three links), and Email 4 PASS","Every FAIL line quotes exact words from that email in quotation marks (for example \"Your Starter plan\" or one of Email 3's extra links)","Each line is only a PASS or FAIL - Cx - \"quote\" verdict (an 'Email N:' label is fine), with no extra commentary, scores, headings or summary"]}