{"id":"eval-dev-judge-json-unknown-sql-column","audience":"developers","category":"evaluation","track":"evals","technique":"constraints","title":"Write a JSON judge that catches made-up SQL columns","free":false,"task":"Your text-to-SQL assistant sometimes queries columns that don't exist, and you want an LLM judge for that one failure mode only, not speed or correctness. Write the judge prompt in exactly 1 code block of at most 30 lines, using the placeholders {schema} and {sql}. It must make the judge return only JSON like {\"pass\": true, \"reason\": \"...\"}, and when pass is false the reason must name the unknown column or table. Keep text outside the code block under 40 words.","materialLabel":"Schema, sample outputs and team notes (attached to your prompt automatically)","material":"Pinecrest Freight - AskData assistant (text-to-SQL for the ops team)\n\nSCHEMA (Postgres)\nshipments(id, customer_id, origin_city, dest_city, shipped_at, delivered_at, weight_kg, status)\ncustomers(id, name, region, signup_date)\ndrivers(id, full_name, home_depot, hired_at)\n\nSAMPLE OUTPUTS FROM LAST WEEK\nQ: \"Late deliveries to Denver in October\"\nSQL: SELECT id FROM shipments WHERE dest_city = 'Denver' AND delivered_at > shipped_at + interval '3 days' AND shipped_at >= '2026-10-01';\n(fine)\n\nQ: \"Email list for customers in the West region\"\nSQL: SELECT customer_email FROM customers WHERE region = 'West';\n(broken: customer_email doesn't exist, query errors in prod)\n\nQ: \"Which drivers deliver the heaviest loads?\"\nSQL: SELECT d.full_name, AVG(s.weight_kg) FROM drivers d JOIN shipments s ON s.driver_id = d.id GROUP BY d.full_name;\n(broken: shipments has no driver_id)\n\nTEAM NOTES\n- Failure modes we've seen: (a) unknown columns/tables, (b) missing date filter so queries scan everything, (c) wrong timezone on shipped_at, (d) SELECT * on big tables.\n- Plan: one judge per failure mode. This ticket is (a) only. Someone else owns (b)-(d).\n- Judge output gets parsed by our CI script with JSON.parse, so any text around the JSON breaks the build.\n- Ravi: \"can we just have one judge give a 1-10 quality score?\" Team said no, scores drifted last quarter.","rules":[{"type":"codeBlocks","count":1},{"type":"maxCodeLines","max":30},{"type":"maxProseWords","max":40}],"mistakes":[{"title":"One judge for everything","why":"The AI happily writes a judge that checks every failure mode at once, which makes its verdicts hard to trust or debug.","fix":"Say: \"Judge only unknown columns or tables. Ignore speed, dates and style.\"","checks":["r0"]},{"title":"A score instead of a verdict","why":"AI judges default to 1-10 scores, which drift and can't gate a build.","fix":"Say: \"Return only JSON with two keys: pass (true/false) and reason.\"","checks":["r1"]},{"title":"A reason that says nothing","why":"Without guidance the judge writes reasons like 'query has issues'.","fix":"Add: \"If pass is false, the reason names the unknown column or table.\"","checks":["r2"]},{"title":"Forgetting the placeholders","why":"The AI pastes this week's schema into the prompt, so the judge breaks when the schema changes.","fix":"Say: \"Use {schema} and {sql} as placeholders.\"","checks":["r2"]},{"title":"An essay around the prompt","why":"The AI explains how judges work and adds usage tips and extra examples in more code blocks.","fix":"Say: \"Exactly 1 code block, at most 30 lines, under 30 words outside it.\"","checks":["codeBlocks","maxCodeLines","maxProseWords"]}],"promptHints":[{"label":"Limits it to one failure mode","patterns":["only","one failure","single failure","nothing else"]},{"label":"Asks for JSON pass/reason","patterns":["json","\"pass\"","pass.{0,20}reason"]},{"label":"Uses placeholders","patterns":["\\{schema\\}","\\{sql\\}","placeholder"]},{"label":"Makes the reason name the column","patterns":["name.{0,30}(column|table)","which column","quote"]},{"label":"Limits the size","patterns":["1 code block","one code block","\\d+\\s*lines","\\d+\\s*words"]}],"hint":"Judges you can trust check one thing and return something a script can parse. Tell the AI the single failure mode, the exact JSON shape, and what the reason must contain.","rubricLabels":["The judge checks only whether the SQL uses a table or column missing from the schema (it does not also grade date filters, timezones, SELECT *, speed, style or whether the answer is correct)","Tells the judge to output only a JSON object with exactly two keys, pass (true or false) and reason (a string), with no score, no extra keys and no text around it","Includes the placeholders {schema} and {sql}, and says that when pass is false the reason must name the specific column or table that isn't in the schema"]}