{"id":"eval-dev-grade-ai-commit-messages","audience":"developers","category":"evaluation","track":"evals","technique":"format","title":"Grade 4 AI commit messages against your team's convention","free":false,"task":"Your team's AI writes commit messages, and you're checking 4 against the convention. This sprint, ticket ids changed from 'PL-' to 'PLAT-', so 'PL-' now fails C3. Grade the 4 messages in exactly 4 numbered lines: '1. PASS' or '1. FAIL - C2 - \"exact words from the message\"', naming the first criterion it fails. Plain text, under 80 words.","materialLabel":"The convention and 4 AI commit messages (attached to your prompt automatically)","material":"platform-api - commit message convention (CONTRIBUTING.md, 2025)\n\nC1: Starts with a type: feat:, fix: or chore:\nC2: The summary uses the imperative mood (\"add\", not \"added\" or \"adds\").\nC3: Ends with the ticket id in brackets, like [PL-123].\n\nAI-GENERATED COMMIT MESSAGES (Nov 5)\n1. \"fix: handle empty cart in checkout total [PLAT-412]\"\n2. \"feat: added retry to webhook sender [PLAT-388]\"\n3. \"chore: bump eslint to 9.2 [PL-401]\"\n4. \"Update README with local setup steps [PLAT-415]\"\n\nNotes\n- The release script groups commits by type, so a missing type breaks the changelog.\n- Someone asked whether emoji prefixes are allowed. Not decided.","rules":[{"type":"numberedItems","count":4},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Missing the id change","why":"The convention still shows PL-, so the AI passes message 3.","fix":"Tell it: \"Ticket ids are now PLAT-. PL- fails C3.\"","checks":["r0"]},{"title":"No evidence behind a FAIL","why":"The AI writes 'wrong mood' without showing the word.","fix":"Say: \"Quote the exact words from the message that fail it.\"","checks":["r1"]},{"title":"Rewrites instead of grades","why":"The AI rewrites each commit message to fix it.","fix":"Show the shape: \"2. FAIL - C2 - \\\"quote\\\"\", no rewrites.","checks":["r2","numberedItems"]},{"title":"Getting markdown back","why":"AI defaults to code formatting and bold verdicts.","fix":"Ask for \"plain text, no bold, no code blocks.\"","checks":["plainText","maxWords"]}],"promptHints":[{"label":"Shows the line shape","patterns":["FAIL - C","PASS","one line per"]},{"label":"Asks for a quote as evidence","patterns":["quote","exact words","evidence"]},{"label":"Gives the id change","patterns":["PLAT-","PL-","ticket id"]},{"label":"Says exactly how many","patterns":["\\b(4|four)\\b.{0,25}(line|message)","exactly \\d+"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"When a convention changes, an AI grader keeps using the written one. Give the change, show the one-line shape, and ask for the quote behind each fail.","rubricLabels":["Grades message 1 PASS, message 2 FAIL on C2 (\"added\"), message 3 FAIL on C3 (old PL- id), and message 4 FAIL on C1 (no type prefix)","Every FAIL line quotes exact words from that message in quotation marks (for example \"added retry\", \"[PL-401]\" or \"Update README\")","Each line uses only the shape 'N. PASS' or 'N. FAIL - Cx - \"quote\"' with no suggested rewrites, explanations or summary"]}