Balancing security and safeguards in military applications of AI
The Anthropic-DOD conflict mainstreamed concerns over fully autonomous weapons, or “killer robots,” and intrusive surveillance. As a result, several members of Congress introduced legislation intended to regulate the DOD’s use of AI, particularly for lethal autonomous weapons systems. Sens. Elissa Slotkin (D-MI), Kirsten Gillibrand (D-NY), Adam Schiff (D-CA), Mark Kelly (D-AZ), Christopher Coons (D-DE), and Jack Reed (D-RI), as well as Reps. Don Beyer (D-VA), Tom Barrett (R-MI), and Sara Jacobs (D-CA), have all introduced versions of regulatory legislation, though none of these bills is expected to be considered for a vote in this Congress given the limited legislating time left in the year.
Both the House and Senate versions of the fiscal year 2027 National Defense Authorization Act (NDAA), on the other hand, contain several of the concepts in the above legislation, along with a variety of provisions regarding the responsible use of AI for defense purposes. Because the NDAA authorizes military programs and expenditures that are core to national security on an annual basis, Congress has successfully passed an NDAA for more than 60 consecutive years. The NDAA thus provides a ready, bipartisan vehicle through which legislators can authorize and safeguard the use of AI. Lawmakers should use it to pass provisions that ensure appropriate human control over AI-enabled weapons; preserve human responsibility in the chain of command; outline appropriate testing criteria; and mandate study and reporting on incidents that may risk troop safety, national security, or law-of-war violations.
Why is congressional action needed now?
The Anthropic-DOD incident contributed to fears over AI safety that have only grown since that time. Recent disclosures from whistleblowers and senior leaders at frontier AI companies have raised the prospect of catastrophic and potentially existential harms, including scenarios involving loss of control over increasingly capable frontier AI models. September polling from POLITICO indicates that a majority of Americans believe “there’s a real risk AI will destroy humanity,” and a plurality “favor pausing AI development.” September polling from Quinnipiac University reflected similar concerns, finding that 91 percent of American adults believe that it is either “very important” or “somewhat important” for the United States to “establish guardrails for AI systems” (though the poll did not ask about military applications specifically).
At the same time, fears around “killer robots” should not obscure the critical role that AI will and should play in defense. Advancements in AI are already changing the battlefield. Russia has swiftly adapted to incorporate task-specific AI throughout its operations in Ukraine and is developing fully autonomous capabilities in its unmanned systems. China, a peer military competitor to the United States, views AI-enabled capabilities as critical to achieving a “world-class” military status by the middle of the century; the People’s Liberation Army is rapidly integrating AI into operations, and Beijing is investing accordingly. While the United States may have more advanced frontier AI models, matching the rapidly advancing nature of adversaries’ capabilities in using AI for military purposes requires research, development, and investments today to deter and counter attacks in the future. Yet it is not only to preserve a military edge that AI is important for defense: AI, with appropriate safeguards, could contribute to mitigating civilian harm or enabling defense applications that can save lives.
But just as fear of autonomy should not encourage an overly restrictive policy framework, neither should fear of adversary competition lead to an overly permissive one. Unfortunately, this is the approach that Secretary of Defense Pete Hegseth is pursuing at the DOD, where he has focused almost exclusively on acceleration. He has sought to create an “‘AI-first’ warfighting force” but has given insufficient consideration to the guardrails necessary to do so safely. While AI may bring benefits to the military, moving too quickly to develop and field systems has risks: loss-of-control incidents, where a rogue, weaponized AI agent may redefine a mission outside the parameters of commander intent, or incidents in which AI-enabled decision-making (such as targeting decisions) may intentionally or unintentionally contribute to violations of the laws of armed conflict. For instance, the United States accelerated its targeting procedures to facilitate a staggering number of strikes in the first hours and days of Operation Epic Fury in Iran in February 2026. After one of these strikes hit a girls’ school and killed 157 civilians, anonymous U.S. officials reported that overreliance on AI—specifically, Palantir’s Maven Smart System, the DOD’s AI-enabled software—likely contributed to more rushed targeting and review processes and may have contributed to the incident.
In June 2026, President Donald Trump issued a presidential memorandum that criticized “undue bureaucracy that hampered the pace of AI adoption” for national security purposes and mandated the acceleration and integration of AI. This included revising DOD Directive 3000.09 on “Autonomy in Weapon Systems,” the DOD’s policy on lethal autonomy. The policy was last significantly updated in 2023 and currently requires “appropriate levels of human judgment over the use of force” by autonomous and semi-autonomous weapons systems, as well as clear and specific criteria for testing and review. While an update to the policy has not yet been made public, the United States recently worked with Russia to strip regulations from proposed international rules governing autonomous weapons, a concerning sign that the administration is not taking seriously enough the need for guardrails.
The NDAA is the best vehicle for advancing AI and establishing guardrails
These developments underscore the imperative for Congress to legislate military applications of AI effectively. The NDAA provides the best vehicle for doing so, both because of the must-pass, bipartisan nature of the legislation and because Congress will need to consider the NDAA in short order before the end of the year. As it does so, Congress has an opportunity both to authorize programs that can give the United States a strategic technological and military advantage and to place commonsense guardrails around the development and fielding, whether in training or in battle, of advanced technologies. As the technology and its military applications rapidly progress, there is an imperative to ensure that the military can adapt accordingly, while mitigating the potential for technological advances to outpace safety considerations. This means specific statutory direction and greater oversight over how the DOD is defining and operationalizing appropriate human control over AI systems, particularly where these are purpose-built to act with some degree of independence from continuous human supervision. This should include instituting specific criteria for testing and review prior to fielding systems, as well as ensuring that accountability is preserved within the chain of command once systems are fielded.
As Congress now works to reconcile the House and Senate versions of the FY 2027 NDAA, it should preserve or bolster language that mandates testing for safety, creates reporting requirements and procedures for adverse incidents, and preserves command and control over autonomous AI agents. This should include:
- Ensuring appropriate human control. Section 1647 of the Senate NDAA, S. 4784, codifies requirements for “appropriate human oversight” to ensure that systems are utilized in accordance with legal and safety requirements. The section defines such oversight as the ability to detect and avoid adverse behavior and to disengage or deactivate systems that operate illegally or outside the bounds of commander or operator intent. It also explicitly prohibits an “autonomous capability” from making the decision to launch or detonate a nuclear weapon, as well as certain AI-enabled domestic surveillance not otherwise permissible under the Constitution. Congress should preserve these safeguards while removing language suggesting that direct human oversight should diminish simply as AI and autonomous technologies mature.
- Preserving human responsibility in the chain of command. Section 1524 of the House NDAA, H.R. 8800, requires a review and update of DOD Directive 3000.09, as well as a report to Congress. This section should incorporate requirements from Section 1653 of S. 4784 and clearly mandate requirements for human responsibility and accountability in the design, testing, fielding, operation, and command and control of AI systems.
- Outlining performance and safety criteria for testing. Several provisions in both H.R. 8800 and S. 4784 deal with requirements for testing AI systems prior to fielding, though these do not stipulate appropriate testing standards. Language in these sections should include assessment criteria such as reliability and robustness; traceability; risk of civilian harm or other violations of the laws of armed conflict; and efficiency of human-machine interaction in realistic simulations. Where applicable, testing should ensure that performance measurably exceeds that of appropriately trained human operators under similar conditions. For instance, Section 834 of H.R. 8800 establishes an acquisition preference for systems that have already deployed in conflict zones and have a “Technology Readiness Level of 7 or above,” but prior fielding should not replace the requirement for adequate testing and review.
- Creating whistleblower protections and reporting channels for adverse incidents. Section 1502 of H.R. 8800 (and a part of Section 1647 of S. 4784) creates a centralized incident and vulnerability reporting program mandating the tracking, analysis, and remediation of incidents in which the DOD’s AI systems jeopardize safety or act outside of operator intent. Some provisions in Section 1647 also include monitoring and reporting mandates and important whistleblower protections. Congress should preserve both the program and protections, as well as stipulate reports to relevant committees with information about safety breaches or national security risks in the interest of transparency.
- Authorizing research and education on safety implications of artificial general intelligence. Although not a guardrail per se, Section 240 of H.R. 8800 includes an initiative to study the national security implications of artificial general intelligence (AGI), the “AGI Preparedness Initiative”; directs the creation of preparedness plans in the event of an AI-related national security incident; authorizes an AI education program for general and flag officers; and requires reports to Congress on these activities. Understanding the national security risks of AGI is an urgent matter, and one that must involve coordination within and outside the Department of Defense—including with the Department of Homeland Security and the intelligence community—to assess risk and inform policy and safety measures as the technology develops.
Conclusion
Rapid advancements in AI bring great potential for the U.S. military to defend the nation more effectively, with greater precision. But any rapid technological advancements require safeguards, particularly when it comes to new AI models that may operate outside of human control. While the federal government will need to grapple with broader concerns around the risks of AI and AGI, increasing autonomy and greater automation in military use carry specific, emergent risks, and lawmakers should address these now—while they have the opportunity. Lawmakers should ensure that the FY 2027 NDAA addresses both the potential for harm resulting from poorly designed automated targeting at the weapon or platform level and loss-of-control scenarios involving autonomous systems, as well as seek to mitigate potential violations of the laws of armed conflict resulting from any AI-enabled targeting decision. Effective legislation would allay public concerns about the use of AI in military applications, and creating proper guardrails would allow for the pursuit of continued research, development, and integration while reducing risks to service members, civilians, and strategic stability. This would allow the U.S. military to maintain technological superiority against adversaries, preserve its ability to deter advanced attacks, and safeguard the laws of war.
Acknowledgments
The author would like to thank Frank Kendall and Adam Conner at the Center for American Progress, as well as Sofia Guerra of Win Without War, for their expert review and feedback on this article.