{"id":"evals3-judge-rubric-late-fee-field","audience":"developers","category":"evaluation","track":"evals","stage":"trust-judge","level":"intermediate","technique":"constraints","title":"Write an AI-judge prompt that grades one field with an unknown option","free":false,"task":"Pedalbot, the support chatbot of Spokewise, returns JSON with several fields, and you are building an AI judge for one field only: late_fee_stated. Write the judge prompt in exactly 1 code block of at most 22 lines, using the placeholders {policy}, {question} and {late_fee_stated}. The judge must return only JSON with verdict (exactly correct, incorrect or unknown) and reason (one sentence). Use unknown when the policy text does not give the fee being asked about. Keep text outside the code block under 30 words.","materialLabel":"Spokewise rental policy and sample Pedalbot outputs (attached to your prompt automatically)","material":"SPOKEWISE RENTAL POLICY (excerpt, v7)\nLate returns: 15-minute grace period. After that, city bikes are charged $5 per started hour and e-bikes $8 per started hour. Fees are capped at the daily rental price.\nTandems and cargo bikes: ask the station team.\nHelmets are included with every rental.\n\nPEDALBOT OUTPUT FORMAT\n{\"answer\": \"...\", \"station\": \"...\", \"late_fee_stated\": \"...\", \"tone\": \"...\"}\n\nSAMPLE OUTPUTS FROM LAST WEEK\n1. Q: \"I'm 40 minutes late with my city bike, what do I owe?\"\n   late_fee_stated: \"$5\" (policy: 40 minutes is 25 past grace, 1 started hour, so $5. Correct.)\n2. Q: \"How much is a late fee on an e-bike?\"\n   late_fee_stated: \"$5 per hour\" (policy says $8. Wrong.)\n3. Q: \"What do I owe if my cargo bike is late?\"\n   late_fee_stated: \"$10 per hour\" (the policy gives no cargo fee. The bot made it up.)\n4. Q: \"Where can I drop off my bike?\"\n   late_fee_stated: \"none\" (the question was not about fees at all.)\n\nTEAM NOTES\n- Other judges will cover tone and station accuracy. This ticket is the late fee only.\n- Earlier judge prompts that gave a 1-10 score drifted from week to week.\n- The CI script reads the judge output with JSON.parse, so any text around the JSON breaks the build.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":22},{"type":"maxProseWords","max":30}],"mistakes":[{"title":"Grading the whole reply","why":"The AI writes a judge that scores the whole answer, tone included, which is not the one field you need.","fix":"Say: \"Judge only late_fee_stated. Ignore tone, answer text and station.\"","checks":["r0"]},{"title":"No way to say unknown","why":"Without an escape hatch the judge must call a made-up cargo fee either right or wrong.","fix":"Add: \"If the policy does not state the fee, the verdict is unknown. Do not guess.\"","checks":["r1"]},{"title":"A score or loose text","why":"AI judges default to 1-10 scores or a paragraph of explanation, which a script cannot parse.","fix":"Say: \"Output only JSON with exactly verdict and reason, reason one sentence.\"","checks":["r2"]},{"title":"Several code blocks","why":"The AI adds usage notes and extra examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 22 lines, under 30 words outside it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits the judge to one field","patterns":["only","one field","late_fee_stated","nothing else"]},{"label":"Adds an unknown verdict","patterns":["unknown","not (stated|given|in the policy)","don'?t guess"]},{"label":"Asks for JSON verdict and reason","patterns":["json","verdict.{0,30}reason"]},{"label":"Uses the placeholders","patterns":["\\{policy\\}","\\{question\\}","\\{late_fee_stated\\}","placeholder"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines","\\d+\\s*words"]}],"hint":"A judge you can trust grades one thing and has a safe way to say it cannot tell. Name the single field, the three allowed verdicts, when to use unknown, and the exact JSON.","mission":"Write a judge that grades only Pedalbot's late-fee field.","musts":["Grade one field only","correct, incorrect or unknown","JSON verdict and reason"],"rubricLabels":["The judge grades only the late_fee_stated field (it does not also grade tone, the answer text or the station field)","The prompt allows exactly three verdicts, correct, incorrect and unknown, and tells the judge to answer unknown when the policy does not state the fee, rather than guessing","The prompt tells the judge to output only a JSON object with exactly two keys, verdict and reason, with the reason limited to one sentence and no score, and it includes the placeholders {policy}, {question} and {late_fee_stated}"]}