{"id":"evals3-pick-the-right-fix-per-failure","audience":"product-managers","category":"evaluation","track":"evals","stage":"fix-prompt","level":"intermediate","technique":"context","title":"Match each Pedalbot failure to the right fix","free":false,"task":"Spokewise has five possible prompt fixes (defined in the material) and four failures from Pedalbot. Write exactly 4 numbered lines, one per failure, each like '1. Failure A: fix name - reason'. Use only the fix names from the material, use no fix twice, and keep each reason under 10 words. One fix will not be used. Plain text, under 90 words.","materialLabel":"The fixes and failures (attached to your prompt automatically)","material":"SPOKEWISE - PEDALBOT FAILURE REVIEW (Nov 6)\n\nTHE FIVE FIXES\n- Allow \"I don't know\": tell the bot it may say it doesn't know when the documents don't answer the question.\n- Quote first: the bot copies the passages it will rely on out of the documents, then answers.\n- A quote per claim: each statement in the answer is followed by the quote that backs it.\n- Verifying second call: a second model call re-checks the finished answer against the documents.\n- Documents-only: the bot may use only the pasted documents, never its general knowledge.\n\nTHE FOUR FAILURES\nA. Q: \"Does the e-bike battery have a warranty?\" The rental terms never mention batteries. Bot answered \"Yes, 2 years.\"\nB. Q: \"What happens if I return the bike late and it is damaged?\" Bot gave a 3-part answer. All parts exist in the terms, but it merged two clauses into one sentence, and reviewers can't tell which clause backs which sentence.\nC. Q: \"How much does a flat tire repair cost?\" Bot said \"Most shops charge $20 to $30.\" The Spokewise fee sheet in the documents says $12.\nD. Q: \"What is the lost-key fee?\" The terms are 60 pages. Bot confidently gave the damaged-lock fee from a nearby clause, and there is no sign of which part it read.\n\nNOTES\n- Ravi: \"A second call doubles our cost per reply.\"\n- All four failures were seen on the same terms document.\n- The fix list was written by the team before the review.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":90}],"mistakes":[{"title":"Using the wrong fix for A","why":"The AI picks the verifying second call because it sounds safest.","fix":"Say: \"A missing answer needs permission to say I don't know.\"","checks":["r0"]},{"title":"Mixing up B and D","why":"Both involve documents, so the AI swaps a quote per claim and quote first.","fix":"Say: \"B: reviewers can't trace sentences. D: it never showed what it read.\"","checks":["r1"]},{"title":"Reusing a fix","why":"The AI picks the same fix for two failures.","fix":"Say: \"Use each fix at most once.\"","checks":["r2"]},{"title":"Long reasons and markdown","why":"AI writes a paragraph per failure with bold fix names.","fix":"Ask for \"plain text, each reason under 10 words.\"","checks":["numberedItems","plainText","maxWords"]}],"promptHints":[{"label":"Asks for one fix per failure","patterns":["one fix","each failure","per failure"]},{"label":"Says use each fix once","patterns":["once","no fix twice","different"]},{"label":"Says to use the listed names","patterns":["names? from","only the fix","from the (list|material)"]},{"label":"Limits the reason","patterns":["under \\d+ words","\\d+\\s*words"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Each fix cures one kind of failure. Read what actually went wrong in each case (no answer, untraceable claims, outside knowledge, wrong section) and pick the fix built for that.","mission":"Pick the best fix for each of four failures.","musts":["Four numbered lines","Use each fix once","Reason under ten words"],"rubricLabels":["Failure A is matched to Allow \"I don't know\" and Failure C to Documents-only","Failure B is matched to A quote per claim and Failure D to Quote first","Each of the four lines uses a different fix name from the list, the Verifying second call is not used, and each reason is under 10 words"]}