{"id":"evals2-choose-explain-cheaper-isnt-cheaper","audience":"managers","category":"evaluation","track":"evals","stage":"choose-model","level":"intermediate","technique":"audience","title":"Explain to a manager why the cheaper model costs more overall","free":false,"task":"Dana Whitfield, Spokewise's ops manager, asked why you want to pay 3x per chat for Quill-L. Reply to her by email, plain text, under 120 words, with no jargon: a line \"Subject: [under 8 words]\", then \"Hi Dana,\", a first sentence that gives the answer, one math line each for Quill-S and Quill-L (per 1,000 chats, wrong answers included), a line \"Saving: ...\" for a month, and a last line \"Ask: ...\" asking for her approval.","materialLabel":"Dana's message and the numbers (attached to your prompt automatically)","material":"From: Dana Whitfield, Ops Manager\nSubject: Why pay triple?\n\nTeam, I saw the request to move Pedalbot from Quill-S to Quill-L. It's three times the price per chat. Why would we do that? I need something I can take to finance, in plain words.\n\nNumbers from the Nov test (1,000 chats each)\n\nQuill-S\n- Price: $3 per 1,000 chats\n- Wrong answers: 18%\n\nQuill-L\n- Price: $9 per 1,000 chats\n- Wrong answers: 3%\n\nWhat a wrong answer costs us\n- A support agent has to find and fix it: $2.50 of staff time each.\n\nVolume\n- About 20,000 chats a month.\n\nOther\n- Quill-S is faster by about half a second.\n- Last year's budget line for Pedalbot was $400 a month in model fees.","rules":[{"type":"plainText"},{"type":"maxWords","max":120}],"mistakes":[{"title":"Defending the price per chat","why":"The AI argues Quill-L is 'better quality' without counting what wrong answers cost.","fix":"Say: \"Compare total cost per 1,000 chats, including $2.50 per wrong answer.\"","checks":["r1"]},{"title":"Jargon for a non-technical reader","why":"The AI mentions accuracy benchmarks, tokens and evals.","fix":"Add: \"Dana is not technical: plain words, no jargon.\"","checks":["r0"]},{"title":"Missing the monthly saving","why":"The AI stops at the per-1,000 figures and never scales to a month.","fix":"Add: \"Saving: the monthly difference at 20,000 chats.\"","checks":["r2"]},{"title":"Long email with no ask","why":"The AI writes a polite, long note with no clear request.","fix":"Say: \"Under 120 words; last line Ask: approve the switch.\"","checks":["maxWords","r2"]},{"title":"Formatted email","why":"The AI uses bold headings and bullet symbols.","fix":"Say: \"Plain text, no bold.\"","checks":["plainText"]}],"promptHints":[{"label":"Names the reader","patterns":["dana"]},{"label":"Counts wrong answers","patterns":["wrong","\\$2\\.50"]},{"label":"Asks for the math","patterns":["math","show","x \\$"]},{"label":"Asks for Saving and Ask lines","patterns":["saving","ask:","approve"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]},{"label":"Says no jargon","patterns":["no jargon|plain words|plain language|non.?technical"]}],"hint":"A manager needs the total cost, not the price per chat. Tell the AI to count staff time for wrong answers, show the math, and finish with one clear ask.","mission":"Convince a manager the bigger model costs less.","musts":["Subject, Hi Dana, answer first","Math for both models","Saving and Ask lines"],"rubricLabels":["The email has a Subject line, then \"Hi Dana,\", and the first sentence says Quill-L is cheaper overall once wrong answers are counted, with no technical terms such as tokens or benchmarks","The two math lines are $3 + 180 wrong x $2.50 = $453 per 1,000 chats for Quill-S and $9 + 30 wrong x $2.50 = $84 per 1,000 chats for Quill-L (equivalent working is fine)","A line starting \"Saving:\" gives $7,380 a month at 20,000 chats ($453 - $84 = $369 per 1,000, times 20), and the last line starts \"Ask:\" and asks Dana to approve the switch"]}