{"id":"eval-pm-golden-answer-subscription-pause","audience":"product-managers","category":"evaluation","track":"evals","technique":"context","title":"Write the golden answer for a subscription pause question","free":false,"task":"You're writing test cases for your app's in-app assistant, and this pause question needs a golden answer. Since Nov 1, the longest pause is 2 months (it was 3), and the help article hasn't been updated. Write exactly 3 numbered lines: '1. Answer: ...' (a direct yes or no), '2. Option: ...' (what they can do instead), and '3. How: ...' (the steps). Plain text, under 60 words.","materialLabel":"The test question and the help article (attached to your prompt automatically)","material":"TEST CASE #18 for the Fieldnote (journaling app) in-app assistant\nUser question: \"I'm traveling for 3 months. Can I pause my Premium subscription for the whole trip?\"\n\nHELP ARTICLE: \"Pausing your subscription\" (last updated in July)\n- You can pause Premium for up to 3 months at a time.\n- While paused, you keep your entries but can't use Premium features like audio notes.\n- To pause: Settings > Subscription > Pause, then pick how long.\n- When the pause ends, billing restarts automatically on the same plan.\n- You can also cancel anytime from the same screen.\n\nTest-writing notes\n- Assistant answers in testing were 100+ words and often offered a free month, which it can't do.\n- Golden answers are the shortest complete correct answer we grade the assistant against.","rules":[{"type":"numberedItems","count":3},{"type":"plainText"},{"type":"maxWords","max":60}],"mistakes":[{"title":"Using the old limit","why":"The help article still says 3 months, so the AI says yes.","fix":"Tell it: \"Since Nov 1, the longest pause is 2 months.\"","checks":["r0"]},{"title":"A no with no option","why":"The AI says no and stops, so the user cancels instead.","fix":"Ask: \"Option: what they can do instead, from the article.\"","checks":["r1"]},{"title":"No steps","why":"The AI says 'you can pause in settings' without the path.","fix":"Ask: \"How: the exact menu path.\"","checks":["r2"]},{"title":"A long reply","why":"The AI writes a paragraph with a free-month offer it can't make.","fix":"Show the shape: \"1. Answer: 2. Option: 3. How:\"","checks":["numberedItems","maxWords","plainText"]}],"promptHints":[{"label":"Gives the new limit","patterns":["2 months|two months","nov(ember)? 1","limit"]},{"label":"Shows the three-line shape","patterns":["answer:","option:","how:"]},{"label":"Asks for an alternative","patterns":["instead","option","alternative"]},{"label":"Keeps to the article","patterns":["only.{0,30}(article|help)","no (free|discount)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"A golden answer has to reflect today's rule, even when the help article is behind. Give the AI the change, show the exact shape, and ask for a real option.","rubricLabels":["Line 1 says no, a 3-month pause isn't possible, because the limit is now 2 months","Line 2 offers pausing for 2 months (optionally with the rest handled another way that is in the article, like cancelling), and nothing offers a free month or discount","Line 3 gives the steps Settings > Subscription > Pause"]}