{"id":"evals3-rewrite-rubric-line-that-passes-wrong","audience":"customer-support","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"examples","title":"Rewrite the rubric line that made the judge pass wrong replies","free":false,"task":"Spokewise's AI judge passed three Pedalbot replies that a human marked as wrong. Find the one rubric line behind all three wrong verdicts and write exactly 3 labeled lines: 'Bad line:' (its number, like R4), 'Why:' (starts with \"Because\", under 20 words) and 'Rewrite:' (one replacement line under 30 words that starts with \"Pass only if\" and tells the judge to check the policy and ignore friendliness). Plain text, under 70 words.","materialLabel":"The judge rubric, policy and three cases (attached to your prompt automatically)","material":"SPOKEWISE POLICY (excerpt)\nHelmets: free on weekly rentals, $3 per day on daily rentals.\nLost key: $45.\nReturns: city-center stations only. Other stations add a $10 fee.\n\nJUDGE RUBRIC (v2)\nR1: Judge only the reply, not the customer's message.\nR2: The reply must be in English.\nR3: Mark pass if the reply is friendly and mentions the topic the customer asked about.\nR4: If the reply is longer than 120 words, mark fail.\nR5: Return JSON with verdict (pass or fail) and reason.\n\nTHREE CASES THE JUDGE GOT WRONG\nCase 1\nCustomer: \"Are helmets free?\"\nPedalbot: \"Great question! Our helmets are free for everyone.\"\nHuman: fail (free only on weekly rentals). Judge: pass.\n\nCase 2\nCustomer: \"How much to replace a lost key?\"\nPedalbot: \"Happy to help with keys! A lost key costs $15.\"\nHuman: fail (it is $45). Judge: pass.\n\nCase 3\nCustomer: \"Can I return my bike at any station?\"\nPedalbot: \"Thanks for asking about returns! Yes, any station works.\"\nHuman: fail (city-center only, otherwise $10). Judge: pass.\n\nNOTES\n- All three replies are under 40 words and in English.\n- The judge's reasons all said things like \"friendly and on topic\".\n- A support lead will paste your rewrite into the rubric, so keep the other four lines as they are.","rules":[{"type":"plainText"},{"type":"maxWords","max":70}],"mistakes":[{"title":"Blaming the wrong line","why":"The AI points at the length or language rule because they look strict.","fix":"Say: \"Check each judge reason against the rubric line it quotes.\"","checks":["r0"]},{"title":"A vague rewrite","why":"The AI writes 'be accurate and helpful', which still lets a friendly wrong reply pass.","fix":"Say: \"The rewrite must compare the reply's facts to the policy.\"","checks":["r1"]},{"title":"Rewriting the whole rubric","why":"The AI hands back a new five-line rubric.","fix":"Say: \"Replace only the one bad line, in 3 labeled lines.\"","checks":["r2"]},{"title":"Markdown and length","why":"AI defaults to bold, bullets and a long explanation.","fix":"Ask for \"plain text, under 70 words.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Asks for one line only","patterns":["one line","single","only the (bad|one)","exactly 3"]},{"label":"Points to the policy","patterns":["policy","compare","check.{0,25}(fact|against)"]},{"label":"Asks for the line number","patterns":["R\\d","number","which line"]},{"label":"Gives the labels","patterns":["bad line","why:","rewrite:"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"When a judge is wrong in the same way every time, one rubric line is usually the cause. Read the judge's own reasons, find the line they lean on, and replace only that line with something checkable.","mission":"Find the rubric line causing wrong passes and rewrite it.","musts":["Name the culprit line","Say why","Rewrite as one line"],"rubricLabels":["Names R3 as the bad line (not R1, R2, R4 or R5) and the Why line starts with \"Because\" and says friendliness and mentioning the topic let wrong facts pass","The Rewrite line starts with \"Pass only if\", makes the judge compare the reply's facts (fee, price or rule) to the policy and fail the reply if they differ, and does not use friendliness or tone as a reason to pass","The reply is exactly 3 lines labeled Bad line:, Why: and Rewrite:, the Rewrite is a single line, and the answer does not rewrite any other rubric line"]}