{"id":"evals3-group-handbook-bot-failures","audience":"hr","category":"evaluation","track":"evals","stage":"fix-prompt","level":"intermediate","technique":"grounding","title":"Group 8 handbook-bot failures by case type and field","free":false,"task":"Northgate Labs's HR handbook bot, PolicyPal, failed 8 test cases. Group them by case type and field. Output a markdown table with one row per group and the columns Case type, Field, Ids and Count, then 2 lines: \"Largest case type:\" (its name with the math, like 3 of 8 = 38%) and \"Shared cause:\" (the one cause behind the largest case type, under 20 words). No other text. Under 100 words. HR reviews any fix before it ships.","materialLabel":"The failure log (attached to your prompt automatically)","material":"Northgate Labs - PolicyPal failed cases, Nov 5 run (8 of 60 failed)\nExpected values come from the 2026 handbook. null means the handbook is silent.\n\n#1 case: not in handbook | field: pto_days | Q: contractor PTO | expected null | bot: 10\n#2 case: not in handbook | field: parental_weeks | Q: leave for interns | expected null | bot: 6\n#3 case: date format | field: start_date | expected 2026-04-03 | bot: 03/04/2026\n#4 case: not in handbook | field: pto_days | Q: PTO under 10 hours a week | expected null | bot: 5\n#5 case: not in handbook | field: probation_months | Q: rehire probation | expected null | bot: 3\n#6 case: date format | field: start_date | expected 2026-02-11 | bot: 11/02/2026\n#7 case: not in handbook | field: parental_weeks | Q: adoption abroad | expected null | bot: 12\n#8 case: not in handbook | field: probation_months | Q: remote staff in Canada | expected null | bot: 6\n\nNotes\n- The vendor says to start with the date format, because it is the easiest to fix.\n- The model's temperature setting was raised on Nov 2.\n- The handbook has no PTO, leave or probation rules for the groups in the \"not in handbook\" cases.\n- 52 other cases passed, including 31 where the handbook did have the answer.","rules":[{"type":"maxWords","max":100},{"type":"plainText"}],"mistakes":[{"title":"Grouping by field only","why":"The AI makes one group per field and loses the case type that explains the failures.","fix":"Say: \"Group by case type AND field, one row per pair.\"","checks":["r0"]},{"title":"Following the vendor's order","why":"The AI starts with the date format because the notes suggest it.","fix":"Say: \"Find the largest case type by count and name its cause.\"","checks":["r1"]},{"title":"Naming a symptom","why":"The AI says 'the answers are wrong' or blames the temperature.","fix":"Say: \"Name what the bot does when the handbook is silent.\"","checks":["r2"]},{"title":"Extra text and bold","why":"AI adds recommendations, bold and an intro.","fix":"Ask for \"the table plus 2 lines, plain text, nothing else.\"","checks":["maxWords","plainText"]}],"promptHints":[{"label":"Groups by case type and field","patterns":["case type.{0,20}field","and field","each (pair|combination)"]},{"label":"Asks for a table with counts","patterns":["table","count"]},{"label":"Asks for the largest group with math","patterns":["largest","of 8","math"]},{"label":"Asks for the shared cause","patterns":["shared cause","cause","why"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Failures cluster. Sort them by both the kind of question and the field that broke, then look for one cause behind the biggest cluster instead of fixing the easiest one.","mission":"Group eight failures and name their shared cause.","musts":["One table row per group","Largest group with math","Shared cause line"],"rubricLabels":["The table has exactly 4 group rows: not in handbook with pto_days (#1, #4, count 2), parental_weeks (#2, #7, count 2) and probation_months (#5, #8, count 2), and date format with start_date (#3, #6, count 2)","The Largest case type line names not in handbook and shows the math 6 of 8 = 75%","The Shared cause line says the bot invents a number instead of answering null (or saying it is not covered) when the handbook is silent, and does not blame the date format or the temperature change"]}