CASE STUDY · A TEACHABLE MOMENT
Pull up a chair. Today’s case study doesn’t come from a fireground, and nobody died. That’s exactly why it belongs in front of a class.
If you’ve sat through a post-incident analysis, you know how they tend to go. Somebody draws the timeline on the whiteboard, and before long somebody else asks who screwed up. A good instructor shuts that question down before it gets comfortable, because it’s the wrong one. People make mistakes. They always have, and they always will. The question that saves lives is how a mistake got past every layer that was supposed to stop it.
A near miss is the cheapest lesson this profession ever gets. Nobody is being buried, no family is waiting on an investigation report, and nobody in the room has to defend a friend. We can look at the thing straight. This one comes from the military rather than the fire service, and it involves artificial intelligence rather than a ladder or a hose line. Stay with me anyway. By the end, you’ll recognize every piece of it from your own calls.
What happened
On September 18 (today) CNN reporters Katie Bo Lillis and Zachary Cohen published an account of an episode from this spring, during the war with Iran, drawn from four sources familiar with it. Here is the chain of events as they reported it.
A special operations command analyst was working with intelligence reporting on the cargo manifest of a Chinese ship in the Middle East, which reporting had come from U.S. Special Operations Command Pacific in Hawaii. The analyst put questions about it to an AI chatbot. The chatbot combined publicly available information with classified signals intelligence held by the government and concluded that the ship was carrying components for a nuclear weapons program.
Then the analyst turned to AI a second time, using it to shape those findings into a standard intelligence report, the kind of product military officials trust, and sent it out. The report circulated across the military.
The military moved. Plans were drawn up to intercept the ship. According to CNN’s sources, armed service members were getting ready to board it, and military aircraft were already in the air. Only shortly before the operation did officials take a harder look at the report, learn that it had been produced with AI’s help, and find that the chatbot had gotten the cargo wrong. One source told CNN the report was completely false and had come close to starting a war. Boarding a Chinese vessel on the strength of it could have spiraled into armed conflict between the two countries.
What we don’t know
A good case study is honest about its gaps, so let’s lay them out before going further.
CNN could not learn what the ship was actually carrying. It isn’t clear whether the chatbot was a commercial product or one built by the government. U.S. Special Operations Command Pacific and the Pentagon did not respond to CNN’s request for comment. The sources are anonymous, which is normal in national security reporting but worth saying out loud. And we don’t know who caught the error, or what made them look twice. That may be the most important thing we don’t know.
I’ve seen people online trying to guess which company’s AI was involved. I’m not going to guess, because it matters far less than the fact that it happened at all. Every one of these tools can produce a confident wrong answer. Swap the brand and you haven’t fixed a thing.
The model
Most of you have seen the Swiss cheese model, even if you didn’t know it by that name. It comes from the late psychologist James Reason, whose work on human error has been taught everywhere from cockpits to operating rooms. His 2000 paper in the British Medical Journal is still the standard citation.
Picture a row of Swiss cheese slices standing on edge. Each slice is a layer of defense: a procedure, a piece of equipment, a person doing a job, a supervisor checking the work. Every slice has holes, because no layer is perfect. Most of the time, a hole in one slice is covered by solid cheese in the next. Disaster comes when the holes line up and a hazard passes straight through.
Reason drew a distinction every officer should carry around. Some holes are active failures, the unsafe acts of the people at the sharp end, whether they’re holding a nozzle, a radio or a keyboard. Others are latent conditions, built into the system by decisions made upstream, often long before and far away, that sit quietly until the day they line up with somebody’s bad moment. He also described two ways of dealing with error. The person approach blames the individual. The system approach asks what conditions made the error likely and which defenses failed to catch it. Only the second one makes the next crew safer.
I’ve leaned on this model in my own instructor training, and I’ll keep leaning on it, because it pulls the conversation away from blame and toward the layers.
Walking the slices
The ship report mapped onto James Reason’s Swiss cheese model. Five layers failed; the sixth held with aircraft already airborne. Slice labels are the author’s analysis of CNN’s reporting.
Slice one is the tool itself. AI chatbots can produce answers that sound authoritative and are simply wrong. The industry calls it hallucination, and it’s no secret. One of CNN’s sources said it hasn’t been an isolated problem across the intelligence community since these tools spread through government. That makes it a known hole, the same way anyone trained on a thermal imager knows it reads surface temperature and can’t see through a wall. A known hole isn’t a failure yet. It becomes one when nobody plans for it.
Slice two is the analyst’s own check. The chatbot’s conclusion could have been held up against the reporting it was built on. Nothing in CNN’s account suggests anyone did that until the very end. This is the active failure at the sharp end, and it’s where most people want to stop, because it hands them somebody to blame.
Slice three is the one that should keep you up at night. The analyst didn’t use AI once. The analyst went back and used it a second time, to package the flawed conclusion as a finished intelligence report. Up to that moment, the mistake lived in one person’s workspace, where one person could still have caught it. After it, a private error had become an institutional fact. The first use was a machine being wrong. The second was a human choosing to carry the machine’s answer forward without confirming it. That’s where a hole in one slice became a tunnel through several.
Slice four is the format. A standard intelligence report carries weight precisely because it’s standard. People downstream trust it because it looks like every other report that turned out to be sound, and that trust is supposed to be earned by the process behind the format. Here the process wasn’t there, and by CNN’s account it took digging to discover that. The people who received it weren’t weighing a chatbot’s guess. They were reading an intelligence report, and they had every reason to believe it.
Slice five is the system around all of it, and this is where the latent conditions live. In January, Defense Secretary Pete Hegseth released an Artificial Intelligence Acceleration Strategy to speed up the military’s adoption of the technology, and the memo announcing it describes putting leading AI models in the hands of the department’s three million civilian and military personnel at every classification level. Multiple U.S. officials told CNN the effort is decentralized, with different parts of government using different tools under different orders and safety standards, and no single standard for verifying what those tools produce. CNN also reported that some older intelligence officials, including ones who broadly support AI, see it pushing analysts to produce and disseminate intelligence faster, and several sources said younger analysts who grew up with these tools are more inclined to trust them without question. None of those conditions caused this error by itself. All of them made it more likely, and the analyst decided none of them.
Slice six held. Shortly before the operation, somebody dug deeper into the report and found the problem. We don’t know who, and we don’t know why. We do know when: with aircraft in the air and armed troops preparing to go aboard a Chinese ship. The system didn’t work. It got lucky at the last possible moment, and those are not the same thing.
False knowns
Here’s the teaching point I’d write on the board and leave there all day.
An unknown makes people careful. When a crew doesn’t know what’s behind a door, they slow down, check the heat, sound the floor and keep a way out. A false known does the opposite. It makes people confident, and confidence is what launches aircraft and puts a boarding team over the rail.
You’ve seen the fireground version. A bystander swears everyone got out of the house. Another insists somebody’s still inside. A dispatcher relays that a caller reports possible entrapment, and three radio transmissions later it has quietly become confirmed entrapment. Each version is only as good as whoever checked it, and every time information changes hands, it looks a little more official and gets checked a little less. That’s what happened to the ship report. It gained authority at every step without ever gaining accuracy.
The rule for your people is easy to say and hard to live by: a report isn’t a fact until somebody confirms it, no matter how official it looks when it reaches you.
The arithmetic
Run the numbers on this one. Checking the chatbot’s conclusion against the original reporting would likely have cost somebody an hour, maybe less. Skipping that check came close to costing a war with a nuclear-armed nation, and in a war, the bill goes to sailors, soldiers and civilians who never saw the report and never had a say in whether anyone checked it.
We live with the same arithmetic. A 360 costs you a minute or two, and every officer reading this knows what skipping it has cost our profession. The price of verifying is small, certain and paid right now by you. The price of not verifying is rare, enormous and paid later by somebody else. That imbalance is exactly why people skip the check, and exactly why no system can depend on every person getting it right every time.
A fair hearing for speed
The people pushing this technology forward aren’t fools, and their reasoning deserves a fair hearing. Officials argue that AI lets the military make battlefield decisions faster, and that the United States can’t afford to fall behind in case it one day has to fight China or another adversary that could otherwise stay a step ahead. The fire service has adopted plenty of technology in the name of speed, and much of it has made us better at the job. Thermal imagers, computer-aided dispatch and vehicle locators all earned their place.
But none of them earned it without training on what they can’t do. Speed only helps if you’re headed in the right direction. One of CNN’s sources summed it up better than I can: “AI allows you to get to a bad idea faster.” And bad ideas, moved fast enough and trusted enough, get innocent people killed, including our own.
This is coming to our house
AI tools are already finding their way into public safety work, from drafting reports to supporting dispatch to writing grant applications. More of them will show up at the firehouse door, most of them sold on the promise of saving time. They may well save it. The question every chief and training officer should be asking now, before the tool arrives, is simple: when this thing is wrong, and it will be, which slice catches it?
In the interest of full disclosure, I use AI tools in my own research and writing, including for this newsletter. The rule I hold myself to is that every figure gets checked against a source before it runs, and when I get something wrong, the correction gets published. That isn’t a boast. It’s the lesson of this case, applied to my own desk.
For discussion
Training officers, here are the questions I’d put to a class after walking through this case.
1. Draw the slices yourself. What was the last point at which one person, acting alone, could have stopped this?
2. The analyst used AI twice. Which use did more damage, and why?
3. What did the finished report tell its readers about how much to trust it, and what did it leave out?
4. Somebody dug deeper at the last minute, and we don’t know who or why. What would your department have to change to make that person the rule instead of the exception?
5. Think about your last working incident. What “known” drove a decision that nobody on scene had actually confirmed?
6. Where does your department already act on a reading, a report or a tool’s output without checking it: a CAD address, a pre-plan, a thermal image, a mutual aid company’s all-clear?
The bottom line
The last slice held this time. Next time it might not, and the time after that, there might not be anyone left to dig deeper. We have to stop treating the whole system as a fail-safe that it isn’t and never has been. Every slice is somebody’s job. Keep the holes in yours as small as you can, and never assume the slice behind you will catch what you let through.
Fred R. LaPoint is Fire Chief and paramedic of Stronach Township Fire Department in Manistee County, Michigan, with over 50 years in emergency services. A Vietnam-era Coast Guard veteran and Team Rubicon disaster responder with more than three dozen deployments, he writes Behind The Alarm and Guardians of Truth.
Sources
1. Katie Bo Lillis and Zachary Cohen, “Exclusive: US military had close call after using AI for false intelligence report, sources say,” CNN, September 18, 2026. https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship
2. James Reason, “Human error: models and management,” BMJ 2000;320(7237):768–770. doi:10.1136/bmj.320.7237.768
3. U.S. Department of War, Artificial Intelligence Strategy for the Department of War, January 2026, as quoted in source 1. https://media.defense.gov/2026/Jan/12/2003855671/-1/-1/0/ARTIFICIAL-INTELLIGENCE-STRATEGY-FOR-THE-DEPARTMENT-OF-WAR.PDF
This article first appeared in Behind The Alarm, the newsletter of Fred R. LaPoint, Fire Chief Paramedic of the Stronach Township Fire Department, on September 18, 2026. Read the original.