Code Isn't a Coup
Both superabundance and apocalypse rest on the same mistaken leap, from powerful technology to a self-directed agent we can no longer stop.
The timeline is up in arms about AI nationalization.
This past Friday President Trump announced he was considering having the government take equity stakes in the largest AI companies, with the proceeds routed back to the public as dividends. No stranger to controversy, breaking tradition, or pursuing strategies historically considered anathema, the President has already directed the government to purchase about ten percent of Intel, ostensibly on the public’s behalf.
Never one to sit out a discussion about wealth redistribution, even Bernie Sanders has proposed a bill for fifty percent federal ownership of the AI megalabs in question. It is not every day that Trump and Sanders reach for the same lever.
In the hallowed words of Oscar Wilde, life imitates art far more than art imitates life. These forecasts have become the genre through which we read the present, and we’ve stopped asking which kind of film we’re in.
AI 2027 and Situational Awareness
Nationalizing frontier AI labs was predicted more than a year ago by a scenario titled AI 2027, a detailed, month-by-month narrative forecast of how AI could progress from today’s agents to superhuman AI and superintelligence by around 2027. Leopold Aschenbrenner’s Situational Awareness, published in 2024, also predicted nationalization, claiming that AI would eventually become a matter of national security.
Both sets of authors have been remarkably successful in their predictions thus far, and they deserve credit for the quality of their research. The AI 2027 authors published their own self-evaluation, and other independent websites now exist that attempt to track its predictions one by one. The capability and safety predictions are the ones tracking closest, several of them early, which is exactly the point: the question is not whether these systems keep getting more capable, it is what that capability turns into.
Aschenbrenner was recently profiled in the Wall Street Journal and boasts impressive returns on investments made on the convictions in his manifesto.
But how do we go from powerful technology to nationalization in a country so wedded to free-market capitalism? How is it that both Republicans and Democrats publicly agree it is the right course of action in the midst of such an aggressively polarized political environment?
Superabundance or Apocalypse?
Since the launch of ChatGPT there has been a steady drumbeat of headlines, proclamations, essays, books, and podcasts all pointing to only one of two hyperbolic outcomes for AI: superabundance or apocalypse.
If the headlines are to be believed, we are either headed for paradise or AI kills us all.
Presenting technological revolutions as the difference between life and death is common. Everyone is interested in the future, and clear, simple outcomes are easy to communicate and easy to understand.
Simple, extraordinary, hyperbolic outcomes are also evidently a fantastic fundraising strategy, because it is hard to raise a hundred billion dollars for a merely useful tool.
Even before founding Anthropic, while still at OpenAI, Dario Amodei was wary of releasing GPT-2, judging it too powerful to put out. He has since written the paradise version in Machines of Loving Grace and the abyss version nearly everywhere else. Demis Hassabis, the field’s designated good guy, keeps sounding the alarm too.
The Messy Middle
But clear and simple outcomes rarely materialize, and grappling with the more probable messy middle means entertaining complexity, nuance, and uncertainty. That takes unusual technical literacy and several rare personality traits most people do not have. Much of this technology is genuinely unprecedented, the frontier is moving, development is accelerating, and diffusion is uneven. So very few people have the tools to evaluate a hyperbolic headline, which is exactly what makes the headlines work.
I made this point last October. We are confidently building toward a future we cannot picture. Eric Schmidt and his peers forecast crisply to 2030, maybe 2035, then simply stop, because no one has a framework for the other side. The honest position, the one that does not trend, is that nobody knows.
But nobody knows is not the same as no forecast. The honest forecast is just a less dramatic one, closer to a couple of points of GDP than to heaven or extinction. Declining to pick paradise or apocalypse is not declining to predict. It is declining to predict theatrically.
The set of plausible futures has exploded, the frontier feels almost infinite, and grappling with infinity is how valuations detach from reason and people start invoking God when they mean machine intelligence.
Suspending Disbelief
The primary problem with these frameworks is the massive, albeit subtle, logical leap they require to go from powerful technology to existential threat to humanity.
The leap is a hallmark of science fiction, and this is the scene where the film asks for it. To lose yourself in Battlestar Galactica, Star Trek, Star Wars, Dune, or Ender’s Game, you first agree to stop asking whether the world on screen could exist. The forecasts ask the same of us, and most of us oblige.
To be clear, the fear of runaway adversarial intelligence is rational. We are right to be afraid of runaway rogue intelligence. But it is misdirected, because it rests on an unfounded assumption. Hidden behind all of the rigorous terminology involving FLOPS of compute, orders of magnitude, exponential development curves, and real, observable trends, there is a simple leap: increasingly capable AI begets self-directed AI.
We’re being invited to believe that really, really good software leads to self-directed, autonomous machines that will want to do good things or bad things depending on how well they are aligned with humanity.
At face value the leap is not immediately evident, which is why it is easy to make. Indeed, long-running agents are already effectively self-directed. But they are self-directed in the loose sense we described in Much Ado About Autonomy:
It is true in a loose sense that agents are programs that can act independently. As I wrote a few weeks ago in AI Waves, an agent is a tool that wields its own tools, does its own work, and has the capacity to operate on its own.
Loose autonomy is independence in execution: humans identify objectives and delegate agents to execute against them. It is useful, valuable, and applicable today.
But autonomous agents also implies a stricter, maximalist form of autonomy: an agent originating its own objectives and operating on its own authority.
The strongest version of the other argument does not need the leap at all. It points at a measured trend: the length of a task an agent can finish on its own is doubling every few months, and Anthropic reports its own models now write most of its code. Extend the line, the argument goes, and the human supervisor quietly disappears. But a longer leash is not a different animal. A model that runs unattended for twelve hours is doing delegated work for twelve hours, not choosing its own ends. What is growing is loose autonomy, execution stretched over a longer horizon. Strict autonomy, originating the objective, is a separate claim, and a rising horizon is not evidence for it.
It’s possible that agents develop strict autonomy, but it is not probable, and it is certainly not inevitable. The line between the two is not clean. A delegated goal spawns its own sub-goals, and a system told to improve itself is, in a narrow sense, choosing what the next system optimizes for. Whether that gradient produces an adversary or simply a more capable instrument is the real question, and it deserves its own essay. The leap is not proven either way. It is just routinely assumed in one direction.
AI is getting extremely good, better than humans, at programming, but despite all of the colossal investment and trillion-dollar valuations, good programming does not necessarily generalize to powerfully adversarial self-directed agents. Great programming will have massive implications for many things including science, medicine, math, and software, all of which will greatly impact society, but it does not have to mean that machines become self-directing, and it does not have to follow that AI is an existential threat.
Asymmetric Information
Artificial intelligence is distinct from other transformative technologies in one respect: it empowers both individuals and nation states.
What AI spokespeople claim when they present the situation is that incredible technology may one day become more powerful than humans. That was true, for example, of nuclear weapons.
Individuals cannot wield nuclear weapons, however, whereas they can use AI to help develop bioweapons, which is why OpenAI and Anthropic, alongside a roster of scientists, signed a letter this month urging lawmakers to tighten screening of the synthetic DNA sequences that could be used to build them.
Asymmetric warfare is a good analogue too. Whereas much of nation-state military spending is focused on increasingly powerful and expensive weapons development, low-cost, unmanned drones powered by advanced AI and deployed by small groups have completely and irreversibly redefined modern warfare, as demonstrated in Ukraine and the Middle East.
The Individual Matters Most
Which brings us back to the government. Perhaps governments are more concerned with nationalizing AI so that they are equipped to protect against all enemies, individual or collective. Concentrated, already-dangerous capability exists, and governments have decided it is already strategically decisive.
Investing in Intel was ordinary industrial policy, a strategy to own a piece of the infrastructure buildout. Reaching for the model companies themselves, on the other hand, is more direct-to-consumer: whether the models become autonomous, conscious, or rogue does not matter.
We’ve been led to believe that the primary threat involved in AI is rogue intelligence, Terminator-style. But it is much more likely to be rogue humans who learn to wield AI in novel, dangerous, and unpredictable ways.
The difference is not cosmetic. If the danger is a rogue machine, the response is to control the model: align it, cap it, build the kill switch, perhaps own the weights outright. If the danger is a rogue human holding the machine, the response is to control access and the chokepoints: who can point the tool, at what, and through which supplier of compute or synthetic DNA. Misdirected control is not a smaller problem than the one we are bracing for. It is the same problem aimed at the wrong target.
By the same token, AI-enhanced operators with good intentions are now equipped to do things that were not possible before. As an investor and an optimist, I am focused on this last reality, but it is just as important not to underwrite paradise as a business plan. Many AI-skeptics believe in the technology and doubt the valuation. It is possible that OpenAI, Anthropic, and the rest build genuinely transformational technology without becoming enduring businesses. The dot-com build-out was real, most of the companies that ran it did not survive it, and the fiber they laid is still in the ground. Transformational and enduring are different words.
If we eschew the hyperbole, AI is a transformative technology. Period. Through that prism you can see where value actually accrues, and the dark fiber is the warning. The rails get overbuilt, so owning raw capacity is not the same as capturing value. What lasts is the scarce layer: the work of improving the models, the verification that makes heterogeneous capacity trustworthy, and the operators who wield it all best. The boom always lays down too much supply and too little of that.
We are surrounded by narratives that jump from capability to civilizational destiny in a single bound. Life imitates art, but we have been bracing for the wrong genre. The blockbuster ending is the part that needs the suspension of disbelief. The story actually unfolding is the docudrama: the people this technology makes powerful, the people it makes dangerous, and the fight, already starting, over which is which.