{"id":"evals2-choose-cost-per-completed-task","audience":"analysts","category":"evaluation","track":"evals","stage":"choose-model","level":"intermediate","technique":"verification","title":"Compute cost per completed task, wrong answers included","free":false,"task":"Tern Logistics wants the true cost of each model answering one shipment-status question. For each model, work out call_cost (input and output tokens at the listed prices), failure_cost (failure rate times the $0.60 a person costs to fix a wrong answer) and cost_per_task (call_cost plus failure_cost). Output one CSV code block with the header model,call_cost,failure_cost,cost_per_task and 3 rows in the order given, dollars as plain numbers with no $ sign. After the block, one line: \"Cheapest per completed task: [model]\".","materialLabel":"Model prices and test results (attached to your prompt automatically)","material":"Tern Logistics - shipment-status bot, model comparison\n\nEvery attempt uses 2,000 input tokens and 500 output tokens, for all models.\nPrices are per 1 million tokens.\n\nModel | Input price | Output price | Wrong answers\nLark-S | $0.50 | $2.00 | 35%\nHeron-M | $1.00 | $4.00 | 20%\nCondor-L | $3.00 | $12.00 | 4%\n\nHow wrong answers are handled\n- Every attempt is billed, right or wrong.\n- A wrong answer is caught and fixed by a person, which costs $0.60 each time.\n- Failure rates come from 500 test questions.\n\nOther facts\n- Volume is about 40,000 questions a month.\n- Lark-S is the model the vendor calls \"budget\".\n- Latency is similar for all three.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":4},{"type":"maxProseWords","max":25}],"mistakes":[{"title":"Ignoring wrong answers","why":"The AI computes only the cost of the call and picks Lark-S.","fix":"Say: \"failure_cost = failure rate x $0.60, added to every task.\"","checks":["r1","r2"]},{"title":"Token-price slip","why":"The AI mixes up per-million pricing and gets the call cost wrong.","fix":"Add: \"Prices are per 1 million tokens: 2,000 tokens x $0.50 / 1,000,000.\"","checks":["r1"]},{"title":"No clean CSV","why":"The AI writes a markdown table or several code blocks.","fix":"Say: \"One CSV code block with the header model,call_cost,failure_cost,cost_per_task.\"","checks":["codeBlocks","maxCodeLines","r0"]},{"title":"Prose around the data","why":"The AI adds an explanation of every calculation.","fix":"Add: \"After the block, only one line: Cheapest per completed task: [model].\"","checks":["maxProseWords","r2"]}],"promptHints":[{"label":"Asks for a CSV block","patterns":["csv","code block"]},{"label":"Gives the header","patterns":["model,call_cost"]},{"label":"Counts failures","patterns":["failure","wrong answer|\\$0\\.60"]},{"label":"Uses the per-million pricing","patterns":["million|1,000,000"]},{"label":"Limits the final line","patterns":["cheapest per completed task","one line|only"]}],"hint":"A model's cost per task is the call plus what its mistakes cost. Give the AI the formulas and the exact CSV header you want.","mission":"Compute each model's real cost per completed task.","musts":["One CSV code block","Failures included in cost","One line naming cheapest"],"rubricLabels":["The CSV header is exactly model,call_cost,failure_cost,cost_per_task, followed by rows for Lark-S, Heron-M and Condor-L in that order","call_cost is 0.002, 0.004 and 0.012 and failure_cost is 0.21, 0.12 and 0.024 for Lark-S, Heron-M and Condor-L (trailing zeros do not matter)","cost_per_task is 0.212, 0.124 and 0.036, and the single line after the block is \"Cheapest per completed task: Condor-L\""]}