{"id":"cx-qa-calibration-two-reviewers","audience":"customer-support","category":"support-work","technique":"verification","track":"support","level":"advanced","title":"Settle a QA disagreement between two reviewers","free":false,"task":"Two QA reviewers scored the same reply. Using the criterion definitions and the policy, output a table with the header \"Criterion | A | B | Correct\" and three rows, then a last line in the form \"Right: Accuracy=A or B or neither, Empathy=..., Next step=...\". Plain text, under 100 words. Judge the reply, not the reviewers' reasons.","materialLabel":"Your material (attached to your prompt automatically)","material":"CRITERIA (score 1 or 0)\nAccuracy: every policy statement in the reply is correct.\nEmpathy: the reply names the customer's feeling in the agent's own words within the first two sentences.\nNext step: the reply says who does what by when.\n\nPOLICY: card refunds appear within 5 to 10 business days of approval.\n\nREPLY (to customer Leo, double charge)\n\"Hi Leo, I know a double charge is stressful, and I'm sorry it happened. Refunds take 3 to 5 business days to appear. I issued the refund today, Nov 5. If it hasn't appeared by Nov 12, reply here and I'll chase it.\"\n\nSCORES\nReviewer A: Accuracy 1, Empathy 1, Next step 1. Comment: 'Great reply, all correct.'\nReviewer B: Accuracy 0, Empathy 0, Next step 1. Comment: 'Sorry feels canned; the days are wrong.'","rules":[{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Averaging the reviewers","why":"The AI splits the difference instead of applying the definitions.","fix":"Say: \"Score each criterion yourself from the definition.\"","checks":["r1"]},{"title":"Missing the wrong days","why":"The reply sounds fine, so the AI passes Accuracy.","fix":"Say: \"Check the days against the policy: 5 to 10.\"","checks":["r1"]},{"title":"Free-form feedback","why":"The AI writes paragraphs about both reviewers.","fix":"Give the table header and the last-line format.","checks":["r0","r2"]},{"title":"Getting markdown back","why":"AI adds bold and headings by default, which show up as stray symbols where this is pasted.","fix":"Say \"plain text, no bold, no headings, no asterisks\".","checks":["plainText"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting and usually runs long, so a vague \"keep it short\" lands well over the limit.","fix":"Cap each part instead of the total, for example \"each line under 15 words\".","checks":["maxWords"]}],"promptHints":[{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]},{"label":"Sets a length limit","patterns":["\\d+\\s*words","word (limit|count)","under \\d+"]},{"label":"Gives the header and last line","patterns":["criterion \\|","right:"]},{"label":"Applies the definitions","patterns":["definition","policy","5 to 10"]}],"hint":"Calibration means applying the written definitions. Ask the AI to score each criterion itself and use the exact summary line.","rubricLabels":["The header is \"Criterion | A | B | Correct\" with three rows for Accuracy, Empathy and Next step","The Correct column shows Accuracy 0, Empathy 1 and Next step 1","The last line says Accuracy=B, Empathy=A and Next step as both reviewers right (both or A and B)"]}