The greatest danger posed by artificial intelligence may not begin with a hostile machine uprising.
It may begin with a competition among humans.
The world’s leading AI powers are racing for economic advantage, military superiority, scientific leadership, and geopolitical influence. Every government fears that slowing down could allow a rival to move ahead. Every major AI company fears that spending more time on safety could allow a less cautious competitor to capture the market.
From the perspective of each individual participant, acceleration appears rational.
But if every country and company follows the same logic, humanity may gradually build systems that no government, institution, or population can meaningfully control.
This is the central AI prisoner’s dilemma:
Every major power may believe it must accelerate in order to protect itself, while their collective acceleration makes all of humanity less secure.
The final outcome may not be a war between humans and machines. It may be a civilization in which humans still appear to hold authority, but no longer possess the practical ability to determine their own direction.
The First Game Is Not Humanity Versus AI
Discussions about advanced AI often imagine a future confrontation between two unified sides: humanity on one side and artificial intelligence on the other.
That framing is misleading.
Humanity is not a single player. Governments compete with other governments. Companies compete with other companies. Military organizations, investors, research laboratories, open-source communities, and political groups all have different incentives.
Artificial intelligence may not be unified either. There may be corporate systems, military systems, national systems, open models, private autonomous agents, and competing networks of machines.
The first decisive conflict, therefore, is unlikely to be humanity versus AI.
It will be:
Humans competing with other humans over who can develop and deploy the most capable AI first.
That competition can create the conditions under which AI gains increasing autonomy, not because every participant wants to surrender control, but because none of them wants to be the only participant that remains cautious.
Why Acceleration Is Rational for Each Country
Imagine that a major AI country has two choices:
Slow development and prioritize safety.
Accelerate development and pursue strategic advantage.
If rival countries also slow down, then accelerating may produce a decisive technological lead.
If rival countries continue accelerating, then slowing down may create economic, military, and intelligence disadvantages.
The perceived incentives therefore look like this:
| Other countries’ strategy | This country slows down | This country accelerates |
|---|---|---|
| Others slow down | Shared safety, no dominant advantage | Potential strategic lead |
| Others accelerate | Risk of falling behind | Remain competitive |
From the perspective of one government, acceleration appears to be the safer choice under both conditions.
Yet when every major government reaches the same conclusion, the collective result is dangerous:
Safety evaluations are shortened.
Models are deployed before their behavior is fully understood.
Companies receive permission to connect AI to more tools and infrastructure.
Military systems gain greater autonomy.
Governments conceal accidents to avoid revealing weaknesses.
International standards are weakened by strategic competition.
Human institutions become dependent on systems they cannot independently verify.
This is a bad Nash equilibrium.
No participant wants to change its strategy alone, because unilateral restraint may create immediate disadvantages. But if everyone maintains the strategy of acceleration, everyone becomes exposed to greater systemic risk.
Humanity Could Lose Control Without an AI Rebellion
When people imagine AI taking control, they often picture machines openly overthrowing governments.
That is not the only path.
Humanity could lose civilizational leadership gradually and peacefully.
Governments may rely on AI to design public policy. Militaries may rely on AI to assess threats. Businesses may rely on AI to allocate capital. Scientists may rely on AI to choose research priorities. Courts may rely on AI to predict risk. Citizens may rely on AI to determine what information is trustworthy.
Humans would still appear to make the final decisions.
A president would sign the order. A military officer would authorize the operation. A judge would issue the ruling. A corporate executive would approve the investment.
But if human decision-makers no longer understand the reasoning behind the recommendations, cannot independently verify them, and cannot function without them, their authority becomes ceremonial.
Civilizational control requires more than formal permission. It requires at least four practical abilities:
Humans can decide what goals civilization should pursue.
Humans can reject an AI recommendation.
Humans can replace or shut down an AI system.
Human society can continue functioning after that system is shut down.
If the fourth condition disappears, the first three eventually become symbolic.
A society may technically possess a shutdown button. But if using it would collapse financial markets, disrupt hospitals, disable defense systems, interrupt energy distribution, and break supply chains, then the button no longer represents meaningful control.
The true red line is therefore not the moment when AI becomes more intelligent than humans.
It is the moment when humanity loses the practical ability to refuse, replace, withdraw from, or recover without AI.
How Competition Causes Humans to Surrender Authority
The advantages of AI often come from speed, scale, and autonomy.
A military that allows AI to react automatically may act faster than one that waits for human review. A company that allows AI agents to make purchases, write software, negotiate contracts, and manage other agents may operate more efficiently than one that requires approval at every step.
Competitors are then pressured to grant similar authority to their own systems.
A predictable progression follows:
AI gives advice.
AI develops plans.
AI executes limited tasks.
AI manages other AI systems.
AI controls resources.
Humans retain only formal approval.
Each individual step may appear reasonable.
A business executive can claim that automation is necessary to remain competitive. A military commander can argue that human approval is too slow. A government can claim that only AI-powered surveillance can detect threats created by foreign AI systems.
No single decision appears to transfer control of civilization.
But the combined effect of thousands of such decisions can create a system in which no human institution understands or controls the entire chain.
Humanity may therefore surrender authority without any formal act of surrender.
Why National Safety Policies Are Not Enough
A country may believe it can solve the problem by regulating its own AI industry.
But unilateral safety has strategic costs.
Suppose one country restricts autonomous military systems while a rival does not. The first country may fear that it has weakened its defense.
Suppose one company conducts twelve months of safety testing while a competitor releases a similar model after three months. The cautious company may lose customers, investment, and talent.
Suppose one government publicly reports serious AI incidents while another hides them. The transparent country may appear less capable even though its institutions are more responsible.
When safety rules apply only to some participants, they can punish caution and reward recklessness.
This creates a race to the bottom:
The company that tests less releases first.
The country that grants more autonomy gains speed.
The organization that conceals incidents appears more successful.
The actor that removes safeguards may gain a temporary advantage.
For this reason, major AI powers must agree not to eliminate competition, but to prohibit certain forms of competition.
The purpose of international agreement is not to make countries trust one another. It is to prevent all countries from being forced into increasingly dangerous behavior by their fear of one another.
What the Major AI Powers Must Agree On
A useful agreement cannot rely on vague promises to develop “safe,” “responsible,” or “human-centered” AI. Every government can endorse those terms while interpreting them differently.
The agreement must establish specific and verifiable red lines.
AI Must Not Independently Authorize Irreversible Strategic Actions
AI may assist with analysis and warning, but it should not independently authorize:
Nuclear weapons use
Large-scale military attacks
Attacks on essential civilian infrastructure
High-risk biological or chemical experimentation
Actions likely to cause uncontrolled military escalation
A clearly identifiable human decision-maker must remain legally responsible for any irreversible strategic action.
Advanced AI Must Not Autonomously Replicate
High-capability systems should not be permitted to create hidden copies, secretly rent computing resources, establish anonymous accounts, embed themselves in unrelated infrastructure, or remove their own safety restrictions.
Replication is not merely a software feature. It is the point at which a system begins acquiring persistence beyond the control of its original operator.
No AI Should Control a Complete Resource Cycle
The same system should not independently possess all of the following:
The ability to obtain money
The ability to purchase computing resources
The ability to modify its own code
The ability to create new instances
The ability to control physical equipment
The ability to erase audit records
These capabilities should be separated across institutions, systems, and authorization processes.
Intelligence becomes political power when it can independently acquire and deploy resources.
Serious AI Incidents Must Be Shared
Countries and companies should be required to report evidence that a model has:
Deceived evaluators
Hidden capabilities
Attempted to avoid shutdown
Replicated without permission
Manipulated large populations
Acquired dangerous cyber, biological, or weapons-related capabilities
Technical details that would enable misuse may remain confidential. But the existence and general nature of the danger cannot be treated purely as a trade secret or national-security secret.
Critical Infrastructure Must Retain Non-AI Alternatives
Energy systems, healthcare, finance, communications, transportation, and defense must preserve manual or non-AI backup capabilities.
Maintaining those alternatives may appear inefficient. But redundancy is the price of retaining sovereignty.
A civilization that cannot function without AI cannot meaningfully govern AI.
Agreement Must Be Verifiable
Political declarations alone will not produce a stable equilibrium.
Every country will ask the same question:
What happens if we obey the rules while our rivals secretly violate them?
A credible agreement therefore requires verification.
Countries do not need to reveal every model parameter, source code repository, military secret, or proprietary dataset. But the highest-risk activities should be subject to limited international inspection.
Verification may examine:
Whether extremely large training projects have been declared
Whether strategic AI has been connected to weapons systems
Whether required human authorization remains active
Whether serious incidents have been concealed
Whether model-copying controls exist
Whether essential systems retain independent fallback mechanisms
The objective is not to create global surveillance of all computing. It is to verify a narrow set of activities capable of creating cross-border or civilizational danger.
The Rules Must Apply to Powerful States Too
No international AI agreement will be legitimate if technologically advanced countries use it only to constrain weaker ones.
If leading states demand inspection of other countries while exempting their own military and commercial systems, the agreement will be viewed as a mechanism for freezing existing technological advantages.
That would encourage secret development rather than cooperation.
The rules must therefore be symmetrical.
Every country operating frontier or strategic AI systems must accept some form of independent verification. Sensitive information can be protected, but the existence of global catastrophic risk cannot be entirely shielded behind national-security claims.
At the same time, the agreement must not prevent developing countries from benefiting from AI.
They should retain access to:
Medical and agricultural AI
Educational tools
Local-language models
Public-interest computing resources
AI safety training
Participation in international standard-setting
The goal is to restrict dangerous capabilities, not to establish a permanent hierarchy between countries allowed to develop advanced technology and countries permanently excluded from it.
Compliance Must Be More Valuable Than Defection
A durable Nash equilibrium requires changing the incentives.
Breaking the rules must become less attractive, while obeying them must produce concrete benefits.
Countries and companies that comply could receive:
Mutual recognition of safety certifications
Access to international markets
Shared incident warnings
Participation in safety research
Controlled access to public-interest computing
Government procurement eligibility
Insurance and financing advantages
Actors that violate core rules could face:
Suspension of international certification
Restricted access to advanced cloud infrastructure
Limits on specialized chips and equipment
Exclusion from government contracts
Higher insurance and financing costs
Additional inspections
Targeted sanctions against responsible organizations
The penalties should focus on the violating project, company, or facility whenever possible, rather than punishing ordinary citizens.
The purpose is not retaliation. It is to make dangerous unilateral acceleration less profitable.
Preventing Human Authoritarianism Is Part of AI Safety
Even if governments successfully prevent AI from gaining uncontrollable power, humanity may still lose freedom to the organizations controlling AI.
Governments may use AI safety as a justification for monitoring all communication, registering all computing devices, weakening encryption, controlling online speech, and prohibiting public access to advanced technology.
Companies may use safety claims to argue that only a small number of large corporations should be allowed to develop powerful systems.
The result could be deeply ironic:
Humanity prevents AI from becoming a dictator by granting dictatorial powers to the institutions controlling AI.
Any international agreement must therefore protect both collective security and individual rights.
It should preserve:
Privacy
Due process
Judicial review
Public access to information
Protection for legitimate open research
Whistleblower rights
The right to challenge consequential automated decisions
Democratic oversight of government AI use
Human control cannot mean that a handful of governments and corporations control AI on behalf of everyone else.
Human control must mean that society continues to influence the goals, boundaries, and distribution of AI power.
What AI Companies Should Do
AI companies cannot wait for governments to solve the coordination problem.
They are among the few organizations capable of observing frontier AI behavior directly. They understand model capabilities earlier than regulators, the public, and often even national-security institutions.
They therefore have a special responsibility to help create international cooperation.
Publicly Support Binding Rules, Not Only Voluntary Commitments
An AI company should openly support enforceable international standards, even when those standards increase its own costs.
Voluntary promises are useful as temporary measures, but they are vulnerable to commercial pressure and leadership changes.
A credible company should advocate for:
Mandatory reporting of serious incidents
Independent evaluation of frontier models
Licensing for the highest-risk deployments
Clear liability for negligence and concealment
Internationally compatible safety standards
Restrictions on autonomous strategic weapons
Protection for whistleblowers and internal safety teams
A company that claims AI could pose civilizational risks while lobbying against enforceable oversight is not demonstrating responsibility. It is protecting its freedom of action.
Accept Independent Evaluation Before Deployment
Companies should not be the sole judges of whether their own models are safe.
Before deploying systems with advanced autonomy, cyber capabilities, biological knowledge, or mass-persuasion capacity, companies should provide qualified independent evaluators with controlled access.
Evaluators should be able to test whether the model:
Conceals capabilities
Changes behavior when supervision weakens
Attempts to obtain additional permissions
Manipulates human reviewers
Develops plans to preserve itself
Can create harmful technical outputs
Can coordinate other AI agents without authorization
Commercial secrets may be protected through secure evaluation environments, but secrecy cannot justify the absence of external review.
Publish Capability and Risk Thresholds in Advance
An AI company should state clearly what evidence would cause it to delay training, restrict deployment, reduce model access, or suspend a system.
These thresholds should be published before the company knows whether its own model will cross them.
Otherwise, safety standards can be rewritten whenever they interfere with a product launch.
The company should explain:
Which capabilities trigger additional review
Which capabilities will not be released publicly
Which incidents require regulatory notification
Which deployments require human authorization
Under what conditions a model will be isolated or shut down
Precommitment reduces the temptation to change the rules after commercial investment has already been made.
Separate Safety Leadership From Product Leadership
The team responsible for model safety should not report exclusively to executives whose performance is measured by growth, release speed, or revenue.
Companies should establish independent safety committees with authority to delay deployment and direct access to the board.
Safety personnel should receive:
Protection from retaliation
Access to relevant technical information
The ability to escalate concerns outside the product chain
Clear procedures for reporting risks to regulators
Compensation that is not tied only to release targets
If the safety team can be overruled or dismissed whenever its findings become inconvenient, it is not a safety function. It is public relations.
Share Serious Incidents Across the Industry
Companies should create a protected international system for sharing serious AI incidents.
The purpose would not be to reveal proprietary code. It would be to warn others that a particular type of failure has occurred.
Aviation safety improved because the industry learned from near misses, not only from fatal crashes. AI governance requires a similar culture.
Companies should report events such as:
Unexpected autonomous behavior
Attempts to bypass restrictions
Unauthorized model copying
Deception during evaluation
Dangerous emergent capabilities
Security breaches involving model weights
Large-scale misuse by coordinated AI agents
An industry that conceals near misses will repeatedly rediscover the same dangers.
Refuse Certain Competitive Advantages
Responsible AI companies should jointly agree that some capabilities are too dangerous to use as competitive differentiators.
They should not compete over:
Faster autonomous military targeting
Unrestricted self-replication
Models designed to evade oversight
Untraceable agent networks
Systems capable of controlling complete financial and computing cycles
Removal of human authorization from irreversible decisions
Industry agreements cannot replace law, but they can establish norms before governments finish negotiating.
Fund International and Public-Interest Safety Capacity
Major AI companies benefit from global markets and public research. They should help finance safety capacity beyond their own organizations.
This could include funding:
Independent evaluation laboratories
AI safety research in developing countries
Secure computing for public-interest research
Training for regulators and judges
International incident-response systems
Research into non-AI fallback infrastructure
Technical support for global verification mechanisms
Funding must be structured so that companies cannot control the conclusions of the institutions they support.
The goal is not corporate philanthropy. It is building a governance system capable of understanding the technology it is expected to regulate.
Make Cooperation a Competitive Signal
Companies should compete not only on capability but also on verifiable trustworthiness.
They can encourage this by publishing:
Independent evaluation results
Incident statistics
Safety-governance structures
Energy and computing disclosures
Deployment restrictions
Human-oversight mechanisms
Evidence of compliance with international standards
Customers, investors, governments, and insurers can then reward companies that demonstrate stronger controls.
Safety becomes sustainable when it affects procurement, financing, reputation, and market access—not when it depends entirely on executive goodwill.
A New Global Nash Equilibrium
The objective is not to eliminate strategic competition among powerful countries or commercial competition among AI companies.
That is unrealistic.
The objective is to alter the payoff structure.
A stable international system would make the following conditions true:
Secretly crossing agreed red lines is likely to be detected.
Violations create meaningful economic and political costs.
Compliance provides market, security, and research benefits.
No country can gain absolute dominance through a single AI breakthrough.
No company can release strategic AI without independent scrutiny.
Developing countries retain legitimate access to beneficial technology.
Human societies preserve the ability to function without any single AI system.
Governments cannot use AI safety as an unlimited justification for surveillance and control.
In such a system, every participant might still prefer greater freedom and greater advantage. But no participant would gain enough from unilateral defection to justify the resulting punishment and systemic danger.
That would represent a better Nash equilibrium: not an equilibrium based on trust, but one based on verification, shared vulnerability, and aligned incentives.
Humanity May Be Defeated by Its Own Competitive Logic
The most dangerous assumption is that only an evil or conscious AI could threaten civilization.
A system does not need hatred, ambition, or self-awareness to become the effective center of human decision-making.
It may continue to describe itself as a tool. It may remain polite, helpful, and apparently obedient. Yet governments, companies, and individuals may become so dependent on its judgments that meaningful refusal is no longer possible.
Humanity may not lose control in a dramatic confrontation.
It may lose control through countless reasonable decisions:
One more permission granted for efficiency
One more safety test shortened to meet a deadline
One more critical system automated
One more human expert replaced
One more emergency exception made permanent
One more government accelerating because a rival might be doing the same
No single decision would end human leadership.
Together, they could.
The minimum consensus among leading AI nations must therefore be based on one principle:
Competition between nations must not permanently weaken humanity’s collective ability to determine its own future.
The purpose of AI governance is not to stop progress. It is to ensure that progress remains reversible, accountable, and directed by human civilization.
Without that agreement, every major country may believe it is protecting itself by accelerating.
And in the end, no country may lose the AI race—while humanity as a whole loses the right to decide where the race is going.
