{"id":"eval-fin-four-tests-ap-invoice-bot","audience":"finance","category":"evaluation","track":"evals","technique":"context","title":"Write 4 test cases for an AI accounts-payable assistant","free":false,"task":"Before your AP team trusts its AI invoice assistant, you want 4 test cases: a duplicate invoice, an invoice in a foreign currency, an invoice with no PO number, and one over the approval limit. Write exactly 4 numbered lines, each 'N. Case | Invoice: short exact description (vendor, number, amount) | Expected: what the assistant does per the AP rules'. Plain text, under 170 words.","materialLabel":"The AP assistant's rules (attached to your prompt automatically)","material":"Ashgrove Partners - AP invoice assistant, rules (pilot starts Dec 1)\n\nThe assistant reads incoming invoices and prepares them for payment. It never sends money; a person approves every payment run.\n\nRules\n- Duplicates: if the vendor and invoice number match an invoice already in the system, it marks \"Possible duplicate\" and does not queue it.\n- Currency: we pay in USD only. Non-USD invoices go to the \"FX review\" queue with the original amount and currency shown. It does not convert.\n- Missing PO number: it routes the invoice to the requester's manager with \"PO missing\" and does not queue it.\n- Amount over $10,000: it queues the invoice but flags \"Controller approval required\".\n- Everything else: queued for the normal Friday payment run.\n\nPilot notes\n- Last month, a vendor emailed the same invoice twice and we paid it twice ($4,300).\n- About 15% of invoices arrive without a PO.\n- The controller (Imani) asked if the assistant can negotiate early-pay discounts. Not in the pilot.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":170}],"mistakes":[{"title":"Tests no one can run","why":"The AI writes 'an invoice in another currency' with no vendor, number or amount.","fix":"Say: \"Each Invoice names a vendor, invoice number and amount.\"","checks":["r1"]},{"title":"Not naming the cases","why":"The AI tests normal invoices and skips the ones that cost money last time.","fix":"List them: \"duplicate, foreign currency, no PO, over $10,000.\"","checks":["r0"]},{"title":"Vague expected results","why":"The AI writes 'flags it for review' instead of the exact queue and label.","fix":"Add: \"Expected uses the rules' exact labels and queues.\"","checks":["r2"]},{"title":"A long test plan","why":"The AI writes a full testing strategy.","fix":"Be exact: \"Exactly 4 numbered lines.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Names the 4 cases","patterns":["duplicate","currency|eur|cad|gbp","po","10,?000|limit"]},{"label":"Asks for specific invoices","patterns":["vendor","invoice number","amount","specific"]},{"label":"Ties expected to the rules","patterns":["rules","expected","label","queue"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(test|case|line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Test cases for money workflows need exact inputs and exact expected labels. Name the risky cases, ask for specific invoices, and tie each expected result to your AP rules.","rubricLabels":["Has exactly the 4 named cases, one each: duplicate invoice, foreign currency, no PO number, and over the $10,000 approval limit","Every Invoice gives a specific made-up vendor, invoice number and amount (the duplicate case names the matching earlier invoice, the currency case names the currency, the limit case is over $10,000)","Every Expected matches the rules: marks 'Possible duplicate' and doesn't queue it; sends to the FX review queue with the original amount and currency and no conversion; routes to the requester's manager with 'PO missing' and doesn't queue it; queues it with 'Controller approval required'"]}