{"id":"eval-fin-grade-ai-expense-categories","audience":"finance","category":"evaluation","track":"evals","technique":"grounding","title":"Grade an AI's expense categories against current policy","free":false,"task":"Your AP team is testing an AI that categorizes expenses, and you're grading 5 of its picks. Since Nov 1, software subscriptions under $50 a month go to Office Supplies, not Software. Write exactly 5 numbered lines: 'N. Correct' or 'N. Wrong - should be [category] - [the rule that applies]'. Plain text, under 90 words.","materialLabel":"The expense policy and the AI's picks (attached to your prompt automatically)","material":"Ashgrove Partners - expense category guide (finance team, updated Sept)\n\nCategories\n- Travel: flights, trains, hotels, rideshares for client or office trips.\n- Meals: food and drinks. Client meals over $75 need the attendee list.\n- Software: any software subscription or license.\n- Office Supplies: paper, toner, desk items, small equipment under $200.\n- Equipment: single items $200 or more (laptops, monitors, chairs).\n\nAI CATEGORIZER - test batch (Nov 4)\n1. \"Figma Pro, monthly\" $15/month -> Software\n2. \"Dell 27in monitor\" $289 -> Equipment\n3. \"Uber to client site, Austin\" $34 -> Travel\n4. \"Staples - toner and printer paper\" $62 -> Equipment\n5. \"Lunch with Bellcrest client team (4 people)\" $96 -> Meals\n\nNotes\n- Controller (Imani) wants the AI live for December close.\n- Last quarter, 9% of manual categories were wrong.\n- The AI's confidence scores were all over 0.9 for this batch.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Using the old guide","why":"The guide still says all subscriptions are Software, and the AI doesn't know about the Nov 1 change.","fix":"Tell it: \"Since Nov 1, subscriptions under $50 a month are Office Supplies.\"","checks":["r0","r1"]},{"title":"No reason for a Wrong","why":"The AI says a category is wrong without saying which rule decides it.","fix":"Ask: \"For Wrong, give the right category and the rule.\"","checks":["r1"]},{"title":"Extra commentary","why":"The AI adds notes about confidence scores and the attendee list.","fix":"Say: \"Only the 5 lines, shaped N. Correct or N. Wrong - should be X - rule.\"","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to a bold table, which won't paste cleanly into your test log.","fix":"Ask for \"plain text, no bold, no tables.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Gives the policy change","patterns":["nov 1","under \\$?50","office supplies"]},{"label":"Shows the line shape","patterns":["correct","wrong - should be","should be"]},{"label":"Asks for the rule","patterns":["rule","why","reason"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"To grade an AI fairly you need the current rules, not just the written guide. Tell the AI what changed, and ask for the right category and the rule behind every miss.","rubricLabels":["Marks 1 Wrong (should be Office Supplies, under-$50 subscription since Nov 1), 2 Correct, 3 Correct, 4 Wrong (should be Office Supplies) and 5 Correct","Each Wrong line names the correct category and the rule that applies (the Nov 1 under-$50 subscription rule for #1, toner and paper are supplies under $200 for #4)","Uses only the line shapes 'N. Correct' or 'N. Wrong - should be X - rule', with no comments on confidence scores, the attendee list or go-live timing"]}