{"id":"evals2-define-label-six-graders","audience":"developers","category":"evaluation","track":"evals","stage":"define-success","level":"beginner","technique":"constraints","title":"Label who should grade each of Pedalbot's 6 output fields","free":false,"task":"Pedalbot's reply has 6 fields. For each one, pick the grader: [CODE] if a program can check it exactly against a known value or pattern, [AI JUDGE] if it is meaning or style that a written rubric can score, or [HUMAN] if a wrong answer risks legal or safety trouble so a person must review it. Write exactly 6 numbered lines in this format: \"1. field_name [LABEL] - reason\", with each reason 8 words or fewer. Plain text, under 110 words.","materialLabel":"Pedalbot reply fields (attached to your prompt automatically)","material":"Pedalbot reply format - v2 draft\n\nEvery Pedalbot answer comes back with these fields.\n\n1. booking_id - must look like SW- followed by 5 digits (example SW-40217) and match the customer's booking.\n2. refund_amount - a dollar figure. Every test case has the correct figure worked out from the refund policy.\n3. escalate - true or false. Every test case has the expected value (true for accidents and injuries).\n4. tone - whether the reply sounds warm and calm and not pushy.\n5. policy_summary - a two-sentence plain-language explanation of the rule that applies. Wording differs from run to run but the meaning should match the policy.\n6. waiver_wording - the paragraph that tells an injured rider what the company is and is not responsible for. A mistake here can create legal exposure for Spokewise.\n\nNotes from the team\n- The first plan was to use one AI judge for every field.\n- About 300 test cases so far.","rules":[{"type":"numberedItems","count":6},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Judging everything with AI","why":"The AI follows the team's first plan and labels every field AI JUDGE.","fix":"Say: \"CODE when a program can check it exactly.\"","checks":["r0"]},{"title":"Missing the risky field","why":"The AI treats the waiver like any other text field.","fix":"Add: \"HUMAN when a wrong answer risks legal or safety trouble.\"","checks":["r2"]},{"title":"Wrong label format","why":"The AI writes Code or Human check without brackets.","fix":"Show it: \"1. field_name [LABEL] - reason\", labels in square brackets.","checks":["r0","r1"]},{"title":"Long reasons","why":"The AI explains each pick in a full sentence.","fix":"Say: \"Each reason 8 words or fewer.\"","checks":["r2","maxWords"]},{"title":"Tables and bold","why":"The AI makes a bold table with extra commentary.","fix":"Say: \"Exactly 6 numbered lines, plain text.\"","checks":["numberedItems","plainText"]}],"promptHints":[{"label":"Defines the three graders","patterns":["code","ai judge","human"]},{"label":"Shows the line format","patterns":["\\[(code|label)\\]","field_name|format","1\\."]},{"label":"Caps reason length","patterns":["8 words|eight words","reason"]},{"label":"Says exactly 6 lines","patterns":["\\b(6|six)\\b.{0,25}(line|field)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting"]}],"hint":"Pick the cheapest grader that is reliable: code for exact checks, an AI judge for meaning and style, a human where being wrong is costly. Show the exact line format you want.","mission":"Choose a grader type for each Pedalbot reply field.","musts":["Exactly 6 numbered lines","Bracketed grader label","Reasons of 8 words"],"rubricLabels":["booking_id, refund_amount and escalate are each labeled exactly [CODE] (in square brackets)","tone and policy_summary are each labeled exactly [AI JUDGE] (in square brackets)","waiver_wording is labeled exactly [HUMAN] (in square brackets), and every line has a reason of 8 words or fewer after a dash"]}