{"id":"evals3-judge-grades-own-model-summaries","audience":"finance","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"verification","title":"Choose between two judges when one grades its own model's work","free":false,"task":"Brightfield Finance Ops uses an AI model to summarize vendor invoices, and two AI judges each graded the same 20 summaries. Humans also checked the 20. Output a markdown table with one row per judge and the columns Judge, Passed, Pass rate and Gap vs humans (in points), then one last line starting \"Recommendation:\" that picks a judge in under 20 words. No other text. Under 70 words.","materialLabel":"Invoice-summary eval results (attached to your prompt automatically)","material":"Brightfield Finance Ops - invoice summary eval (20 summaries)\nThe summaries are written by Model Q. The question for each: does the summary state the right vendor, total and due date?\n\nHuman reviewers (two people, agreed on all 20): 14 summaries good, 6 had an error.\n\nJudge Q (the same Model Q that wrote the summaries): passed 19 of 20.\nJudge R (a different model from another vendor): passed 15 of 20.\n\nNotes\n- Judge Q costs about a third less per run and is faster.\n- Priya (AP lead): \"Q passes 19 of 20, so our summaries are basically solved.\"\n- Judge R flagged 3 of the 6 summaries with errors that Judge Q passed.\n- Summaries feed a draft accrual report. A controller reviews the report before anything is posted.\n- Nothing here is a statement of the company's books; it is a quality check on drafts.","rules":[{"type":"maxWords","max":70},{"type":"plainText"}],"mistakes":[{"title":"Missing the human baseline","why":"The AI compares the two judges to each other and never to the human 70%.","fix":"Say: \"Gap vs humans = judge pass rate minus 14 of 20 = 70%.\"","checks":["r0"]},{"title":"Choosing on cost","why":"The AI picks the cheaper, faster judge because the notes mention it.","fix":"Say: \"Choose on whether the judge grades its own model's work.\"","checks":["r2"]},{"title":"Calling the summaries solved","why":"The AI repeats the 19 of 20 headline.","fix":"Add: \"Compare each judge to the human result.\"","checks":["r1"]},{"title":"Extra text and bold","why":"AI adds an intro, bold and notes.","fix":"Say: \"Table, then one Recommendation line, plain text, nothing else.\"","checks":["maxWords","plainText"]}],"promptHints":[{"label":"Compares to the human result","patterns":["human","14 of 20","70"]},{"label":"Asks for points","patterns":["points?","gap"]},{"label":"Asks for a table","patterns":["table","columns?"]},{"label":"Gives the check for self-grading","patterns":["own (model|work|output)","same model","grades? (its|itself)"]},{"label":"Asks for one Recommendation line","patterns":["recommendation","one (last )?line"]}],"hint":"A judge tends to rate its own model's writing kindly. Compare each judge to the human result, and choose the one that is not grading its own work.","mission":"Pick the judge that does not grade its own work.","musts":["One row per judge","Gap vs humans in points","Recommendation line"],"rubricLabels":["The table has one row per judge: Judge Q 19 of 20 = 95% with a gap of 25 points, and Judge R 15 of 20 = 75% with a gap of 5 points (human reference is 14 of 20 = 70%)","The Recommendation line picks Judge R","The Recommendation gives the reason that Judge Q grades summaries written by its own model (so it rates its own work too kindly), not cost or speed"]}