{"id":"eval-pm-launch-call-by-language","audience":"product-managers","category":"evaluation","track":"evals","technique":"verification","title":"Make a per-language launch call from eval results","free":false,"task":"Your AI support assistant is launching in new languages, and each needs at least 90% on the eval to ship. Japanese was pulled from this launch yesterday, so leave it out. Write exactly 3 numbered lines, one per remaining language, each 'Language: X of Y = Z% vs 90% - Ship or Hold', then a last line 'Summary: ship [languages], hold [languages]'. Plain text, under 80 words.","materialLabel":"The eval results by language (attached to your prompt automatically)","material":"Norland Telecom - AI support assistant, language eval (Nov 3)\nEach language was tested on the same 80 translated support questions, graded pass/fail by native-speaker reviewers.\n\nLanguage | passed | total\nSpanish | 76 | 80\nGerman | 70 | 80\nFrench | 74 | 80\nJapanese | 61 | 80\n\nNotes\n- German failures were mostly formal vs informal \"you\" (Sie vs du) mix-ups.\n- Overall across all four: 281 of 320 = 88%.\n- Marketing already announced \"4 new languages\" in a teaser post.\n- The vendor says Japanese will improve with their next model.\n- Ops can staff a fallback human queue for any language that doesn't ship.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":80}],"mistakes":[{"title":"Including the pulled language","why":"The AI doesn't know Japanese was pulled yesterday, so it grades all four.","fix":"Tell it: \"Japanese is out of this launch. Leave it out.\"","checks":["r0"]},{"title":"Hiding the math","why":"The AI says German is 'a bit low' without the numbers.","fix":"Show the shape: \"Language: X of Y = Z% vs 90% - Ship or Hold.\"","checks":["r1"]},{"title":"Deciding on the average","why":"The AI quotes the 88% overall and holds everything, or ships everything.","fix":"Add: \"Decide each language on its own score.\"","checks":["r2"]},{"title":"A launch memo","why":"The AI writes a long memo with risks and the marketing teaser.","fix":"Be exact: \"3 numbered lines, then one Summary line.\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Leaves out Japanese","patterns":["japanese","pulled","leave out|exclude"]},{"label":"Gives the bar","patterns":["90","bar","threshold"]},{"label":"Asks for the math","patterns":["x of y","math","of 80","show"]},{"label":"Asks for a per-language call","patterns":["ship or hold","each language","per language","summary"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"An average can ship a language that isn't ready. Tell the AI what's in scope, give it the bar, and ask for the math and a call per language.","rubricLabels":["Leaves out Japanese entirely and reports Spanish 76 of 80 = 95% Ship, German 70 of 80 = 87.5% Hold, and French 74 of 80 = 92.5% Ship","Each language line shows the math (X of 80 = Z%) and compares it to 90%","The Summary line says ship Spanish and French, hold German, and nothing uses the 88% overall number to decide"]}