{"id":"eval-dev-api-answer-claims-vs-docs","audience":"developers","category":"evaluation","track":"evals","technique":"verification","title":"Check an AI's API answer against the real docs","free":false,"task":"A coding assistant answered a teammate's question about your payments API, and you want to check it against the docs before it goes in the wiki. Write exactly 5 numbered lines, one per claim, each labeled Supported, Contradicted or Not in docs. For each Contradicted line, add what the docs say in quotation marks. Judge only by the docs excerpt. Plain text, under 110 words.","materialLabel":"The docs excerpt and the AI's answer (attached to your prompt automatically)","material":"DOCS EXCERPT - Ledgerline Payments API v3 (docs.ledgerline.example/v3/refunds)\n\nPOST /v3/refunds\n- Body: payment_id (required), amount_cents (optional; defaults to the full amount).\n- Partial refunds are allowed. You can issue up to 5 partial refunds per payment.\n- Refunds can be created up to 120 days after the original payment.\n- Requires the header Idempotency-Key. Requests without it return 400.\n- Returns 201 with the refund object. Status starts as pending and moves to succeeded or failed.\n\nAI ASSISTANT'S ANSWER (claims numbered by you)\n(1) To refund, POST to /v3/refunds with the payment_id. (2) If you leave out amount_cents, it refunds the full amount. (3) You can do up to 10 partial refunds on one payment. (4) Refunds are allowed for 180 days after the payment. (5) The refund is processed instantly, so the status is succeeded right away.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":110}],"mistakes":[{"title":"Asking 'is it correct?'","why":"The AI says the answer is 'mostly right' and skips the wrong numbers.","fix":"Say: \"Label each claim Supported, Contradicted or Not in docs.\"","checks":["r0","r2"]},{"title":"No proof for corrections","why":"The AI says a claim is wrong but not what the docs say.","fix":"Add: \"For Contradicted, quote the docs in quotation marks.\"","checks":["r1"]},{"title":"Getting a code sample","why":"The AI 'helps' by writing a corrected request in a code block.","fix":"Be exact: \"Exactly 5 numbered lines, one per claim, no code.\"","checks":["numberedItems","r2"]},{"title":"Getting markdown back","why":"AI defaults to bold labels and headings.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Gives exact labels","patterns":["supported","contradicted","not in docs","label"]},{"label":"Makes the docs the judge","patterns":["only.{0,30}(docs|excerpt)","against the docs"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Says exactly how many","patterns":["\\b(5|five)\\b.{0,25}(line|claim)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI answers about APIs often get a limit or a default slightly wrong. Give the AI three labels, make the docs the only judge, and ask for the docs' words when it disagrees.","rubricLabels":["Labels claims 1 and 2 Supported and claims 3, 4 and 5 Contradicted","The Contradicted lines give what the docs say in quotation marks: up to 5 partial refunds, 120 days, and that the status starts as pending","Uses only the three labels Supported, Contradicted and Not in docs, with no rewritten answer, code sample or intro"]}