{"id":"evals2-define-pick-success-line","audience":"product-managers","category":"evaluation","track":"evals","stage":"define-success","level":"beginner","technique":"constraints","title":"Pick the one success definition for Pedalbot a test can check","free":false,"task":"Spokewise's kickoff notes list five ways to define success for Pedalbot. A usable definition passes three tests: a test set can check it, it has a number to hit, and it measures whether the answer was correct (not how it sounds or how fast it is). Write exactly 3 lines: \"Pick: [one letter]\", \"Why: [one sentence under 20 words]\", and \"Rejected: [the other four letters, each with a reason of 3 words or fewer, like D (no number)]\". Plain text, under 70 words.","materialLabel":"Kickoff notes (attached to your prompt automatically)","material":"Spokewise kickoff notes - Pedalbot success definition (Oct 14)\n\nPedalbot is the support chatbot for Spokewise, a bike-rental company. It will answer questions about hours, prices, bookings and refunds from policy sheet v4.\n\nCandidate definitions people brought to the meeting\nA. Customers enjoy chatting with Pedalbot.\nB. Pedalbot sounds friendly and on-brand in every reply.\nC. Pedalbot gives the policy-correct answer to at least 95 of every 100 questions in our 200-question test set.\nD. Pedalbot reduces support tickets a lot this year.\nE. Pedalbot replies in under 3 seconds.\n\nContext\n- The 200-question test set was drafted from policy sheet v4 last week. Every question has a written correct answer.\n- Maya (CEO) said she wants customers to love it.\n- Ravi (ops) said fewer tickets is the real prize.\n- Last month's reply speed averaged 2.1 seconds.\n- The next release is Nov 3.","rules":[{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Picking by gut","why":"The AI explains all five options at length and hedges between C and E.","fix":"Say: \"Pick the one a test set can check, with a number, that measures correct answers.\"","checks":["r0"]},{"title":"A long Why","why":"The AI writes a paragraph of reasoning instead of one short sentence.","fix":"Say: \"Why is one sentence under 20 words.\"","checks":["r1"]},{"title":"Rejections without reasons","why":"The AI lists the losers or writes a sentence for each one.","fix":"Add: \"Rejected lists the other four letters, each with a reason of 3 words or fewer.\"","checks":["r2"]},{"title":"Bold labels and bullets","why":"By default the AI formats a mini report with bold text.","fix":"Add: \"Plain text, exactly 3 lines.\"","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"The AI runs over word limits when it explains choices.","fix":"Cap each part: \"Why under 20 words, each reason 3 words or fewer.\"","checks":["maxWords"]}],"promptHints":[{"label":"States the three tests","patterns":["testable|checkable|can (be )?check","number","correct"]},{"label":"Asks for one pick","patterns":["pick","exactly one|one letter"]},{"label":"Sets the Rejected format","patterns":["rejected","3 words|three words"]},{"label":"Says exactly 3 lines","patterns":["\\b(3|three)\\b.{0,20}line","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"A good success definition can be checked by a test, has a number and is about correct answers. Ask the AI to apply those three tests and give each loser a tiny reason.","mission":"Choose the one testable success definition for Pedalbot.","musts":["Exactly 3 labeled lines","Reasons of 3 words","Plain text"],"rubricLabels":["Line 1 is exactly \"Pick: C\" (no other letter is picked)","Line 2 starts \"Why:\" and is one sentence under 20 words that mentions the 95 of 100 (or 95%) number and checking correct answers","Line 3 starts \"Rejected:\" and covers A, B, D and E (not C), each with a reason of 3 words or fewer, where E's reason is about speed rather than correctness and D's reason says it has no number or is vague"]}