AI intelligence report falsely identifies weapons on Chinese ship, nearly triggers US military operation

According to Security Affairs, an artificial intelligence-generated intelligence report falsely identified weapons components on a Chinese vessel, nearly leading to a US military operation during the Iran war.

An analyst at Special Operations Command Pacific used a chatbot to analyze intelligence regarding a Chinese ship, combining open-source and classified signals intelligence, as first reported by CNN. The AI system produced a false conclusion that the vessel was carrying nuclear weapons components. This erroneous report was then used to generate a formal intelligence report, which quickly circulated through the US military in the spring. Armed boarding teams were prepared, and military aircraft were already airborne, indicating a near-miss for a potential armed confrontation. The report was later discovered to be entirely false, with sources telling CNN it "almost started a war" and could have escalated tensions between the US and China.

This event underscores the dangers of AI hallucinations when verification steps are bypassed, particularly within military decision-making processes. The Pentagon's strategy to accelerate AI integration across its personnel raises concerns about the potential for confidently incorrect AI outputs to bypass human review and lead to critical errors.

Source: Security Affairs