This spring, US officials discovered that the intelligence directing an armed operation targeting a Chinese-owned vessel had been fabricated by an AI chatbot. The operation, which was canceled while military aircraft were in the air, narrowly prevented a potential confrontation with China; CNN reported on Friday. The report was published on September 18, 2026.
A report circulated during the war with Iran claimed that the vessel was carrying components belonging to a nuclear weapons program. The false information was generated after an analyst within Special Operations Command turned to the chatbot to combine open-source data with classified signals intelligence. The bot misidentified the vessel’s cargo manifest. The analyst later used the tool again to turn the erroneous findings into an official-looking summary, and the document was shared through command channels.
The incident occurred as the US military was rapidly adopting AI to speed up decision-making and maintain its advantage over China. The Pentagon says the technology accelerates the “kill chain”; however, insufficient human oversight can allow errors to move up the chain of command.
GovAI researcher and former US Army officer Jake Steckler said it is especially important to understand the uncertainty of LLMs in targeting, intelligence analysis and operational planning. Steckler described the incident as a call to add stronger safeguards, not to avoid AI. Prioritizing speed over safeguards could reduce soldiers’ trust in the systems.
Why it matters
The incident shows that AI has entered military decision-making processes not merely as technical support, but as a factor capable of affecting the chain of command. The failure to subject information produced by systems used in targeting, intelligence analysis and operational planning to human oversight can allow erroneous data to acquire official status and reach decision-makers. This situation highlights the question of how to strike a balance between speed and verification, particularly for soldiers operating in high-risk conditions such as tensions with China and war with Iran. The unresolved issue is how the military will strengthen the safeguards and chain of accountability needed to detect erroneous content while continuing to use AI.