{"id":"eval-dev-judge-json-reply-language-mismatch","audience":"developers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a JSON judge for replies in the wrong language","free":false,"task":"Your support bot sometimes replies in English to customers who wrote in another language. Write an LLM judge for that one failure mode, in exactly 1 code block of at most 20 lines, with the placeholders {user_message} and {reply}. It returns only JSON {\"pass\": bool, \"reason\": str}; a false pass names both languages. Product names, URLs and code snippets in English don't count as switching languages. Under 40 words outside the block.","materialLabel":"Examples from the multilingual rollout (attached to your prompt automatically)","material":"Tessera Cloud - support bot, multilingual rollout (week 2)\n\nEXAMPLE 1\nUser (es): \"¿Cómo cambio la contraseña de mi cuenta?\"\nBot: \"Para cambiar tu contraseña, ve a Settings > Security en el panel de Tessera Cloud.\"\nReviewer: fine\n\nEXAMPLE 2\nUser (de): \"Wie kann ich meine Rechnung herunterladen?\"\nBot: \"You can download your invoice from Billing > Invoices.\"\nReviewer: bad, replied in English\n\nEXAMPLE 3\nUser (fr): \"Mon script échoue avec cette erreur, pourquoi ?\"\nBot: \"Ton script échoue parce que la variable n'est pas définie. Essaie : export API_KEY=...\"\nReviewer: fine\n\nTeam notes\n- Reviewers also flag wrong answers, missing doc links and overly long replies.\n- The judge runs nightly on 2,000 chats and writes JSON lines to a file.\n- Supported languages: English, Spanish, German, French, Portuguese.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":20},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI also checks accuracy and length, so a fail doesn't tell you it was the language.","fix":"Say: \"Check only whether the reply matches the user's language.\"","checks":["r0"]},{"title":"Failing harmless English","why":"A strict judge fails replies that keep English menu names or code.","fix":"Add: \"Product names, URLs and code in English don't count.\"","checks":["r1"]},{"title":"A score instead of a verdict","why":"AI judges default to scores and paragraphs, which break a JSON file.","fix":"Say: \"Return only JSON with the keys pass and reason. A false pass names both languages.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains the judge and adds examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 20 lines, one sentence after it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","ignore","nothing else"]},{"label":"Excuses English names and code","patterns":["product name","code","url","don't count"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses placeholders","patterns":["\\{user_message\\}","\\{reply\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines"]}],"hint":"A language judge compares two texts and nothing else. Say what English is allowed, keep it to one failure mode, and show the JSON shape.","rubricLabels":["The judge checks only whether the reply is in the same language as the user's message (it does not also grade accuracy, doc links, length or tone)","Tells the judge that product or menu names, URLs and code snippets in English don't count as switching languages","Uses the placeholders {user_message} and {reply} and tells the judge to return only JSON with exactly the keys pass and reason, where a false pass names both languages"]}