David Sacks Calls Anthropic's Agent Self-Improvement Warning a Bid for Nationalization. Half of Washington Agrees.

NCT covered Anthropic’s “When AI Builds Itself” paper on June 5. This commentary covers the political and industry reaction that followed.

Former White House AI Czar David Sacks responded to Anthropic’s 10,000-word warning about recursive self-improvement with a post on X that stripped the debate to its structural tension: “Signs you might be trying to get your frontier AI lab nationalized: You compare it to nukes, threaten half of white-collar jobs, warn recursive self-improvement could end humanity, then race ahead anyway. In other words, you want the government to save us from you.”

The accusation landed because the timing is hard to ignore. Anthropic confidentially filed for an IPO days before publishing the paper. The company is pursuing a valuation reportedly near $1 trillion. And the paper itself opens with internal data showing Claude writes 80% of Anthropic’s merged production code, a number the company frames as evidence of danger but which doubles as a competitive moat disclosure.

Two Readings of the Same Paper

The reaction split cleanly into two camps, each with credentialed voices.

The regulatory capture reading. Gary Marcus, professor emeritus at NYU, told Business Insider the proposal is “an incredible, cost-free piece of rhetoric, perfectly timed for the IPO.” He argues Anthropic wants people talking about a pause option “they don’t actually plan to take, and are unlikely to ever take.” Luis Garicano, economics professor at the London School of Economics and former member of the European Parliament, wrote on X that the real target is open-source competition: “The key threat to the profitability of frontier models is open weights. If they scare the hell out of everyone, the natural move will be to forbid them and allow only ‘trusted developers.’” Kylan Gibbs, CEO of Inworld AI and former Google DeepMind, said the playbook is straightforward: tell governments AI is dangerous, and when regulation comes, you are the one they consult.

The governance reading. Former Senator Mitt Romney wrote on X that AI safeguards are “our highest and most urgent national priority,” listing risks from weapons to mass unemployment. Andrew B. Hall, professor of political economy at Stanford, said a coordinated slowdown “would have seemed totally non-credible even pretty recently” but no longer seems farfetched, given recent executive orders, Anthropic’s Project Glasswing expansion, and OpenAI’s own proposal for beefed-up model review.

Noah Giansiracusa, associate professor of mathematics at Bentley University, told Scientific American he does not think the call is genuine. “I don’t think it’s a genuine call to slow down. I think he wants to keep going full speed ahead.” But he also called a pause “literally impossible.” Georgia Tech professor Mark Riedl posted on Bluesky that “the big AI companies are all jumping on the ‘recursive self-improvement’ hype train.”

An Anthropic spokesperson clarified to Business Insider that the company is not calling for a pause now. Instead, it wants leading competitors to have systems in place that would allow one. Given the pace of development, Anthropic wants to study the mechanism before it becomes necessary.

The Biosecurity Dimension

Separately, Ben Buchanan, a former Biden administration cyber policy official now at Johns Hopkins and an adviser to Anthropic, told CBS News’ “Face the Nation” that AI assisting biological weapons development is “not hypothetical.” Buchanan framed AI governance as fundamentally a biosecurity and national security question, not just a labor economics debate. The interview aired the same week as the self-improvement paper, and both Buchanan and former CISA director Chris Krebs discussed whether existing government structures are adequate for regulating AI systems that can operate autonomously.

Where Agent Builders Sit in This

The policy split is not abstract for anyone building on or deploying autonomous agents. The Anthropic paper is explicitly about coding agents, research agents, and autonomous systems that complete multi-hour tasks without human intervention. The internal data it cites, from Claude fixing 800 bugs to agents running open-ended safety research, describes the same category of tool that agent platforms ship to customers.

If Sacks’ deregulation stance prevails, agent frameworks operate in a permissive environment where speed-to-market dominates. If Anthropic’s governance stance gains traction, agent platforms face compliance requirements around audit logging, human oversight triggers, and potentially coordinated deployment limits for frontier-capable agents.

The practical question is which bet you are making by choosing a model provider. Anthropic is building toward a world where regulation constrains who can deploy autonomous agents and under what conditions. OpenAI published its own federal governance blueprint last week proposing a federal agency (CAISI) to monitor recursive self-improvement. The Trump administration’s posture, articulated by Sacks, treats safety rhetoric from frontier labs as a competitive weapon rather than a policy input.

Francesco Bianchi, economics professor at Johns Hopkins, told Business Insider that the risk “might be real, but it is very convenient for a market leader” to be the one defining it. That captures the bind precisely. The question of whether autonomous agents need governance is legitimate. The question of who gets to write the rules is political. Right now, the companies building the agents are the ones proposing the frameworks, and the people who want to keep building without frameworks are the ones in government.

For agent builders, the play is not to pick a side. It is to build audit and oversight infrastructure now, before the policy question resolves, so that compliance is a configuration toggle rather than a rewrite.