{"id":"evals2-define-fix-lopsided-case-set","audience":"analysts","category":"evaluation","track":"evals","stage":"define-success","level":"beginner","technique":"context","title":"Rebuild a 20-case set so it matches Pedalbot's real traffic","free":false,"task":"Pedalbot's current test set has 20 cases, and every one is an easy question. Rebuild it as 20 cases whose mix matches last month's real traffic exactly (scale the traffic percentages to 20). Write a markdown table with the columns Category, Cases and Example input, one row per category in the order listed in the traffic data, then a final line \"Total: 20\". Each example input is a customer message in quotes. Plain text, under 190 words.","materialLabel":"Test set and traffic data (attached to your prompt automatically)","material":"Pedalbot test set review - Spokewise (Nov 3)\n\nCurrent test set: 20 cases\n- 12 on hours and location (\"When do you open?\")\n- 8 on prices and bike types (\"How much is an e-bike for a day?\")\n- 0 on refunds, 0 on damage reports, 0 on upset customers\n- Pass rate on this set: 100%\n\nLast month's real traffic: 1,200 chats\n- Hours and location: 480\n- Prices and bike types: 300\n- Refunds and cancellations: 240\n- Damage and accident reports: 120\n- Angry or off-topic messages: 60\n\nNotes\n- Dev lead: \"We pass 100%, so we're fine.\"\n- Safety lead: \"Accident chats worry me most. I'd put half the tests there.\"\n- Support lead: refund chats reopen twice as often as any other kind.\n- Tests only help if they look like what customers actually send.","rules":[{"type":"maxWords","max":190},{"type":"plainText"}],"mistakes":[{"title":"Overweighting the risky cases","why":"The AI follows the safety lead and puts half the tests on accidents.","fix":"Say: \"Match last month's traffic percentages exactly: 8, 5, 4, 2, 1.\"","checks":["r0"]},{"title":"Keeping only easy cases","why":"The AI adds a few edge cases but leaves the old mix mostly intact.","fix":"Say: \"Scale the traffic counts to 20; every category gets its share.\"","checks":["r0"]},{"title":"Missing the total line","why":"The AI ends with a paragraph of advice instead of a total.","fix":"Add: \"End with exactly: Total: 20.\"","checks":["r1"]},{"title":"Describing, not writing, inputs","why":"The AI writes \"a question about refunds\" rather than a message.","fix":"Add: \"Each Example input is a real customer message in quotes.\"","checks":["r2"]},{"title":"Essay around the table","why":"The AI adds a strategy section and bold headings.","fix":"Say: \"Plain text, only the table and the total line.\"","checks":["maxWords","plainText"]}],"promptHints":[{"label":"Matches the real traffic","patterns":["traffic|percent|%|scale|proportion"]},{"label":"Asks for a table","patterns":["table","columns?"]},{"label":"Asks for example inputs in quotes","patterns":["example input","in quotes|quoted"]},{"label":"Asks for the total line","patterns":["total: 20","total"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"A test set should look like the chats customers really send. Tell the AI to scale last month's traffic mix to 20 cases and ask for a realistic example message per category.","mission":"Rebuild the case set to match real traffic.","musts":["Cases scaled to traffic","Table plus Total: 20","Quoted customer messages"],"rubricLabels":["The table has the headers Category, Cases and Example input, with one row per category in the order of the traffic data, and the Cases column reads 8, 5, 4, 2, 1","The last line is exactly \"Total: 20\"","Every Example input is a realistic customer message in quotes that fits its category (for instance, the damage row is about damage or an accident and the last row is an angry or off-topic message)"]}