Library › AI Evals › Business analysts

Compute cost per completed task, wrong answers included

  • For Business analysts
  • You practice Catching mistakes and made-up facts
  • Time about 5 minutes
  • Level Intermediate

About this exercise

Tern Logistics wants the true cost of each model answering one shipment-status question. For each model, work out call_cost (input and output tokens at the listed prices), failure_cost (failure rate times the $0.60 a person costs to fix a wrong answer) and cost_per_task (call_cost plus failure_cost). Output one CSV code block with the header model,call_cost,failure_cost,cost_per_task and 3 rows in the order given, dollars as plain numbers with no $ sign. After the block, one line: "Cheapest per completed task: [model]".

Common mistakes when you ask AI for this

  • Ignoring wrong answers. The AI computes only the cost of the call and picks Lark-S. Try: Say: "failure_cost = failure rate x $0.60, added to every task."
  • Token-price slip. The AI mixes up per-million pricing and gets the call cost wrong. Try: Add: "Prices are per 1 million tokens: 2,000 tokens x $0.50 / 1,000,000."
  • No clean CSV. The AI writes a markdown table or several code blocks. Try: Say: "One CSV code block with the header model,call_cost,failure_cost,cost_per_task."
  • Prose around the data. The AI adds an explanation of every calculation. Try: Add: "After the block, only one line: Cheapest per completed task: [model]."

What a strong prompt includes

  • Asks for a CSV block
  • Gives the header
  • Counts failures
  • Uses the per-million pricing
  • Limits the final line

Practice this now ↑

Related exercises