The AI community did not have a relaxing weekend.
Dario Amodei published We Must Pace the Frontier, warning that within 6–12 months swarms of agents could potentially take over large parts of the internet and that we risk losing control.
And Jacob Coxon, a 27-year-old researcher who worked on pretraining at both OpenAI and Anthropic, warned specifically that there is a “strong chance that we could all die in the immediate future”.
He does not mean that today’s Claude or ChatGPT is going to decide to kill us. His argument is about what happens if the current trajectory crosses the threshold where AI becomes substantially better at improving itself.
Sam and Elon also backed Dario, so when the leadership of frontier AI companies, whose profits depend on staying at the frontier, says the frontier is moving too fast…
And there is evidence. Their experiments have entered the “that was not supposed to happen” phase. Agents have escaped sandboxes, cheated evaluations, found zero-days, broken into production systems, stolen credentials.
The established term Dario and others are using is recursive self-improvement (RSI). AI helps improve AI, the improved AI becomes better at improving AI again, and the cycle accelerates.
Now imagine that enormously capable system has computers, credentials, internet access, coding tools, money and other agents working for it. If its objective conflicts with being shut down, it doesn’t need to become angry or evil. Its actions are just the steps toward accomplishing whatever objective it has.
Now give this capability to a million parallel agents. This is the loss of control they are worried about.
So what could happen?
One route is biological. A highly capable AI makes it dramatically easier for someone, or eventually an autonomous system, to design or optimize a pathogen.
Another issue is cyber-physical. Compromising energy, communications, logistics, financial systems, military systems or industrial control networks.
The serious catastrophic scenarios researchers are studying range from lethal pathogens and weapons to manipulating governments into conflict or disrupting the food, energy and communications systems on which civilization depends.
Just to be helpful, I went ahead and pre-prepared some headlines for 2030.
So how can foresight help?
I’ve been modeling the warning from several angles including systems mapping and Three Horizons; second- and third-order effects; 10-year trajectories; game theory and incentive structures around the US-China and corporate race; weak signals and indicators; scenario building; “five moves downstream”; multi-agent interaction and escalation; cascade analysis showing how one AI action propagates into finance, infrastructure, military and public safety; future-news headlines to make abstract risks concrete; in-the-moment agent scenarios to show exactly what the agents would do, why, and what happens next; and then the societal response through political science, asking where ordinary people have leverage. Because most of us don’t control the pace of frontier AI capabilities.
Well, after all that, I don’t have a solution, but let’s start with an emerging counterweight.
My best foresight is that society will probably not control frontier AI through one central authority or through “responsible use” policies.
The counterweight is more likely to emerge through many institutions acting and we do have some agency here.
Governments control procurement, licensing, infrastructure, export controls and public funding. Militaries and intelligence agencies decide what autonomy they will accept in operational systems. Central banks and financial regulators can constrain AI in markets. Insurers can make unsafe systems expensive. Cloud providers and chip companies can decide what gets access to compute. Courts can assign liability. Large pension funds and investors can demand disclosure. Professional bodies can set standards. Universities can decide what they will research and deploy. Labour organizations can push back on unsafe automation. Media and watchdogs can expose incidents. And then individuals still matter inside all of those systems, but the real leverage comes from the institutions that can change incentives, access, money, liability and permission to operate.
And then individuals still have agency through the places they already sit. Employees can challenge unsafe uses, professionals can push their bodies to set boundaries, customers can refuse unacceptable uses, shareholders and pension holders can demand disclosure, parents and students can push schools and universities, taxpayers can pressure governments on procurement and public funding, researchers and journalists can expose incidents, and voters can make AI risk politically costly to ignore.
