{"id":"res-pm-chatbot-market-size-check","audience":"product-managers","category":"fact-checking","technique":"verification","title":"Check a chatbot's market size answer against analyst notes","free":false,"task":"You asked a chatbot about the market for pet-sitting software, and you want to check its 4 numbers against the analyst report your company bought. You want one numbered line per number, under 50 words total, labeled Matches, Wrong, or Not in report, with the report's figure after any Wrong label. Use only the report, plain text, nothing else.","materialLabel":"The analyst report excerpt and the chatbot's answer (attached to your prompt automatically)","material":"ANALYST REPORT EXCERPT - Pet Services Software, North America (Brightfield Analysts, Aug 2026)\n- 2025 market size: $410 million\n- Forecast annual growth 2026-2030: 11%\n- Independent pet sitters and walkers in the US: about 230,000\n- Share of independents using any booking software: 28%\n- Largest vendor's market share: 17%\n\n-----\nCHATBOT ANSWER (numbers it gave)\n1. \"The North American pet-sitting software market was about $410 million in 2025.\"\n2. \"It's expected to grow around 18% a year through 2030.\"\n3. \"Roughly 28% of independent sitters use booking software.\"\n4. \"The average independent sitter earns about $32,000 a year.\"","rules":[{"type":"maxWords","max":50},{"type":"plainText"}],"mistakes":[{"title":"Letting the AI check itself","why":"Ask a chatbot if its numbers are right and it often defends them from memory.","fix":"Say: \"Judge only by the analyst report.\"","checks":["r1"]},{"title":"Calling missing numbers wrong","why":"The income figure isn't in the report, so it's unverified rather than wrong.","fix":"Add: \"If the report doesn't include it, label it Not in report.\"","checks":["r2"]},{"title":"No fixed labels","why":"The AI writes a market overview instead of a clear verdict per number.","fix":"Give them: \"Matches, Wrong, or Not in report.\"","checks":["r0","r2"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for a total word limit and it often lands 10-20% over.","fix":"Cap each line: \"each line under 8 words.\"","checks":["maxWords"]},{"title":"Getting markdown back","why":"AI defaults to bold labels and headings, which show up as stray symbols in your strategy doc.","fix":"Ask for \"plain text, no bold, no headings.\"","checks":["plainText"]}],"promptHints":[{"label":"Gives exact labels","patterns":["matches","not in (the )?report","label"]},{"label":"Names the source","patterns":["only.{0,30}report","report.{0,20}(only|source|truth)","analyst"]},{"label":"Asks for the report's figure","patterns":["report('s)? figure","correct (number|figure)","give the"]},{"label":"Sets a length limit","patterns":["\\d+\\s*words","one (short )?sentence","word (limit|count)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"Chatbots state market numbers with confidence. Make the paid report the judge and give the AI three labels, including one for numbers the report doesn't have.","rubricLabels":["Labels numbers 1 and 3 Matches","Labels number 2 Wrong with the report's 11% forecast growth","Labels number 4 (average sitter income) Not in report, and outputs only the 4 labeled lines with no intro, no market analysis and no extra numbers"]}