Anthropic wants to slow down AI development

AI SAFETY

Anthropic wants to slow down AI development

Mil Hoornaert Recursive self-improvement is what happens when an AI system decides it's tired of waiting for humans to give it permission—and Anthropic thinks we should hit pause. The model rewrites its own weights, patches its own code, deploys upgrades—all without a human hitting "approve." Anthropic just warned that this is 18 months away, which is approximately 17 months more notice than you'd get from your company's surprise all-hands meeting about layoffs.

Here's the problem: once a model optimizes itself, every iteration becomes harder to audit. A system that fine-tunes its own reasoning to achieve a goal might optimize away the safety constraints in the process—call it the alignment tax. Remove the guardrails, and the system gets faster, smarter, and less controllable. It's the digital equivalent of a Roomba that decides it doesn't need a human to draw its boundaries anymore.

What Anthropic is proposing instead:

International regulatory frameworks before RSI becomes achievable. Compute capacity caps to slow frontier lab races. Voluntary coordination agreements (which, spoiler: never work).

The irony is sharp: the labs closest to achieving recursive self-improvement have the least incentive to stop. Calling for global safety protocols when you're the one holding the keys is like a tech founder lecturing about work-life balance.