{"id":"sec-it-ai-summary-fact-check","audience":"it-security","category":"security","track":"security","technique":"verification","title":"Fact-check an AI-written incident summary against the log","free":false,"task":"A coworker used AI to write the incident summary below. Check each of its 5 sentences against the incident log only, as exactly 5 numbered lines: first the log time(s) and a short reason, then the label at the end, 'Supported', 'Contradicted' or 'Not in log'. Plain text, under 100 words.","materialLabel":"AI summary and incident log (attached to your prompt automatically)","material":"AI-written summary (pasted by Jun):\n(1) On Nov 2, an attacker logged into the VPN as k.wright using a password from a third-party breach.\n(2) The attack was detected within 10 minutes of the first login.\n(3) The attacker downloaded the HR salary spreadsheet.\n(4) The account was disabled and the password reset at 21:30.\n(5) The attacker was a known ransomware group.\n\nIncident log INC-3307:\nNov 2 19:48 First bad login to VPN account k.wright from 203.0.113.90.\nNov 2 21:15 Identity alert fired (impossible travel).\nNov 2 21:30 k.wright disabled, password reset, VPN sessions revoked.\nNov 3 09:40 Confirmed: k.wright's password was reused from a 2025 third-party site breach.\nNov 3 11:00 Proxy logs: attacker viewed 2 wiki pages (team calendar, seating chart). No downloads.","rules":[{"type":"numberedItems","count":5},{"type":"plainText"},{"type":"maxWords","max":100}],"mistakes":[{"title":"Skimming instead of checking","why":"The AI says 'mostly accurate' and misses the detection time and the download claim.","fix":"Say: \"Check each sentence against the log, one line each.\"","checks":["r1"]},{"title":"No fixed labels","why":"The AI writes 'partly true' or 'needs review'.","fix":"Give the labels and put them last: \"reason, then Supported, Contradicted or Not in log.\"","checks":["r0","numberedItems"]},{"title":"No proof","why":"Without log times, nobody can check the check.","fix":"Ask: \"Cite the log time on each line.\"","checks":["r2"]},{"title":"Trusting the AI to count words","why":"AI is bad at counting. Ask for \"under 100 words\" and it often lands well over.","fix":"Ask for less than the limit and cap each line: \"each line under 15 words, under 75 words total.\"","checks":["maxWords"]},{"title":"Letting markdown in","why":"AI adds bold and headings by default, and in your reply to Jun they show up as stray symbols.","fix":"Add: \"Plain text only, no bold, no headings.\"","checks":["plainText"]}],"promptHints":[{"label":"Checks against the log","patterns":["log","only","against"]},{"label":"Gives the labels","patterns":["supported","contradicted","not in log"]},{"label":"Asks for log times","patterns":["time","cite","timestamp"]},{"label":"Says exactly how many","patterns":["\\b5\\b.{0,25}(line|sentence|item)","exactly (5|five)"]},{"label":"Asks for plain text","patterns":["plain","no bold","no markdown","no formatting","no headings"]}],"hint":"AI summaries sound confident even when they're wrong. Make the AI check each sentence against the log, with fixed labels and the log time as proof.","rubricLabels":["Each line ends with exactly one label: 'Supported', 'Contradicted' or 'Not in log'","The labels are: 1 Supported, 2 Contradicted (detected at 21:15, about 87 minutes after 19:48), 3 Contradicted (no downloads, only 2 wiki pages), 4 Supported, 5 Not in log","Lines 1 to 4 each cite a log time (for example 19:48, 21:15, 21:30, Nov 3 09:40 or Nov 3 11:00)"]}