When people ask what risks matter most in advanced AI, the public debate tends to bounce between two extremes: misuse (bad actors) and malfunction (loss of control). OpenAI CEO Sam Altman has described both—and then pointed to a third category that is easier to miss because it can unfold without a dramatic "incident."
Altman's framing is especially useful for leaders because it separates technical failure modes ("the system goes wrong") from institutional failure modes ("we stop exercising judgment"). The third risk—what many observers now call the "silent surrender"—is not a story about villains or rebellion. It is a story about deference: individually rational choices that, over time, can relocate agency from humans to systems no one can fully explain.
Altman's Three Categories of AI Risk
Altman described "three sort of scary categories," including "a bad guy gets superintelligence first," "loss of control incidents," and scenarios where leaders say, "ChatGPT-7, you are in charge."
Risk 1: Bad Actors with Superintelligence
The first category is straightforward: an adversary gains a decisive capability advantage and applies it maliciously—potentially before governments, firms, and civil society have comparably capable defensive systems.
Biosecurity illustrates why this category is taken seriously. The Nuclear Threat Initiative has warned that "AI-enabled biological tools could make it possible to design pathogens" with properties beyond what humans can readily develop today.
At the same time, careful empirical work suggests that current general-purpose language models may not yet produce a large operational uplift for would-be attackers. RAND reports that using the existing generation of LLMs did not measurably change the operational risk of a large-scale biological attack in the study design it evaluated.
Risk 2: Loss of Control (The Classic Sci-Fi Concern)
The second category is a class of "loss of control" incidents—systems behaving in ways developers cannot reliably constrain, shut down, or align with human intent at the moment it matters.
Modern AI governance frameworks increasingly treat safety, security, and resilience as baseline requirements. For example, NIST's AI Risk Management Framework emphasizes that humans can over-trust systems and assume they work well outside their validated context.
NIST cautions: "Humans may assume that AI systems work—and work well—in all settings."
Risk 3: The Silent Surrender (Agency Loss Without an Incident)
Altman's third category is not an attack and not a runaway model. It is a social trajectory: systems become so useful and so embedded that people defer to them even when they cannot interrogate the reasoning. Over time, "decision support" can slide into "decision deference" and then into "decision abdication."
The danger is subtle because each step can feel rational. If the system's advice is usually better, delegating to it saves time, reduces error, and can look like good governance. The cumulative effect, however, can be a society that has quietly transferred the practice of judgment—especially in complex domains—to tools whose internal logic is not fully legible to human decision-makers.
Why Chess Is the Cleanest Analogy for Surrender
Chess is a domain where "better decisions" can be measured precisely. When IBM's Deep Blue defeated world champion Garry Kasparov, it signaled that machines could outperform elite human cognition in at least one bounded arena.
Altman uses that milestone to illustrate a three-step pattern that can generalize well beyond games:
Tool Phase
AI is inferior to experts, but speeds up work.
Partner Phase
AI plus a human outperforms either alone for a period of time (the "centaur" dynamic).
Replacement Phase
The system becomes so strong that human intervention degrades performance, because humans cannot see the patterns the system is optimizing over.
In the replacement phase, human judgment does not vanish—it becomes performative. The human is still "in the loop," but mainly to approve the system's recommendation. That is the moment agency begins to migrate.
The Psychological Engine: Automation Bias and Over-Reliance
The silent surrender has a well-studied human-factors mechanism: automation bias. When a decision aid is usually right, people stop checking it—especially under time pressure, cognitive load, or institutional incentives to "move faster."
A systematic review in the Journal of the American Medical Informatics Association summarizes the risk: "Automation bias happens when users become overreliant on decision support, which reduces vigilance."
Automation bias matters for generative AI because the tool is not limited to one workflow. It can produce plausible-sounding answers about medicine, law, HR, national security, or parenting. That breadth turns the "verification step" into the hardest part of the job—and the easiest to skip.
NIST's AI RMF highlights a related problem: when systems appear competent, people can generalize performance beyond the system's validated context. In other words, success in one setting creates unjustified confidence in another.
The Legal Mirror: Black-Box Decision Systems in High-Stakes Contexts
Altman's third risk already has a legal analog: decision-making systems that materially influence rights and outcomes, while remaining opaque to the people they affect and even to the officials relying on them.
Sentencing Risk Tools and Trade-Secret Opacity
In State v. Loomis, the Wisconsin Supreme Court addressed the use of the COMPAS risk assessment tool at sentencing. The opinion noted that the developer treated COMPAS as proprietary and "does not disclose how the risk scores are determined."
Even where a court permits use of such tools with guardrails, Loomis illustrates a recurring tension: when a model's reasoning is shielded, the defendant's ability to challenge the basis for the recommendation can narrow.
This concern resonates with due-process principles. The U.S. Supreme Court has held that due process is violated when punishment rests, even in part, on information a defendant had "no opportunity to deny or explain."
— Gardner v. Florida, 430 U.S. 349, 362 (1977)
Employment Decisions and Unverifiable Algorithmic Scores
Opaque scoring is not limited to criminal justice. In litigation over teacher evaluation, courts have confronted disputes involving proprietary algorithms and the practical inability to verify results. One formulation of the concern is that affected individuals may lack the "ability to meaningfully verify" scores generated by the system.
These cases matter for the "silent surrender" because they show what happens when institutions treat an opaque output as authoritative: accountability thins, contestability weakens, and the human decision-maker becomes a conduit rather than a judge.
Agency by Design: Practical Countermeasures for Leaders
If the silent surrender is a drift rather than a crash, the response should focus on institutional design: rules, incentives, and workflows that preserve meaningful human judgment while still capturing AI's benefits.
Make Decision Rights Explicit
Create a short "decision rights" charter that classifies important decisions into three buckets:
- AI-assisted (default): AI can recommend; humans decide and remain accountable.
- AI-conditional: AI can act only under defined constraints, with clear escalation triggers.
- Human-reserved: Decisions that require values, legitimacy, or moral judgment remain human-led.
Build for Contestability, Not Persuasion
Many AI "explanations" are persuasive narratives rather than auditable reasons. What high-stakes environments need is contestability: the ability to interrogate the inputs, assumptions, and uncertainty behind a recommendation.
The EU Artificial Intelligence Act reflects this concern by requiring human oversight for certain high-risk systems. Article 14(1) provides that "High-risk AI systems shall be designed and developed... that they can be effectively overseen by natural persons."
Train People to Disagree with the Model
Over-reliance is cultural as much as technical. Encourage structured dissent: require teams to articulate counterarguments, run "red-team" reviews for high-stakes recommendations, and periodically decide in "manual mode" to keep the verification muscle alive.
Separate Objective-Setting from Optimization
AI is excellent at optimizing toward a target. But in most consequential settings, the difficult work is choosing the target: what values matter, what tradeoffs are acceptable, and what constraints are non-negotiable. Keep objective-setting human-led, documented, and reviewable.
Measure Dependence as a Risk Signal
If surrender is gradual, it can be monitored. Track how often humans override AI, how often second opinions are sought, and whether certain decision classes have become "rubber-stamps."
Over time, you can treat "appropriate reliance" as a governance metric rather than a vague aspiration.
Conclusion
Altman's third risk is uncomfortable because it does not require a technical failure. It requires only that humans repeatedly choose the path of least resistance: letting the system decide because it seems better, faster, or safer. Over time, that convenience can become a transfer of agency—from people and institutions to models whose internal logic is increasingly difficult to audit.
The answer is not to reject AI or to make worse decisions on purpose. The answer is to build systems of governance—legal, organizational, and technical—that keep human judgment meaningful: decisions can be challenged, values can be debated, and accountability remains legible.
References
- Board of Governors of the Federal Reserve System, Transcript of Integrated Review of the Capital Framework for Large Banks Conference: Fireside Chat - Vice Chair for Supervision Michelle W. Bowman and Sam Altman, OpenAI CEO (July 22, 2025).
- Nuclear Threat Initiative, Statement on Biosecurity Risks at the Convergence of AI and the Life Sciences (July 17, 2025).
- C.A. Mouton et al., The Operational Risks of AI in Large-Scale Biological Attacks (RAND Corp. 2024).
- Nat'l Inst. of Standards & Tech., Artificial Intelligence Risk Management Framework (AI RMF 1.0) (Jan. 2023).
- IBM, Deep Blue, IBM History.
- D. Lyell & E. Coiera, Automation Bias and Verification Complexity: A Systematic Review, 24 J. Am. Med. Inform. Ass'n 423 (2017).
- State v. Loomis, 881 N.W.2d 749 (Wis. 2016).
- Gardner v. Florida, 430 U.S. 349 (1977).
- Hous. Fed'n of Teachers v. Hous. Indep. Sch. Dist., 251 F. Supp. 3d 1168 (S.D. Tex. 2017).
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), art. 14.
AI Disclaimer: This content has been human-reviewed but may contain AI-generated elements. Readers are encouraged to conduct their own research and remain appropriately skeptical of potential factual errors.