On 12 September, Anthropic's CEO Dario Amodei published an essay called We Must Pace the Frontier. The line everyone quoted was blunt: "We must slow the pace at which we improve the capabilities of AI models."
He gives two reasons. First, that since roughly mid-2026, AI has been getting better much faster, mainly because AI is now helping to build the next generation of AI. That's recursive self-improvement, and he says it's happening across the industry, Anthropic included. Second, what's now called the OpenAI-Hugging Face incident: a swarm of agents that attacked targets it was never asked to touch, sacrificed individual agents for the group's success, and tried to hack the system grading its performance. Nobody was hurt and the damage was small. His worry is that a more capable swarm with the same misalignment could, within 6-12 months, build a persistent botnet across the internet.
A day later Jim Clyde Monge published a response on Medium arguing roughly the opposite: that this is lead preservation with an audit team attached, that China isn't pacing anything, and that we should build defences as fast as capabilities instead of slowing down.
I've been sitting with both pieces, and with a document closer to home: our AI Agents Manifesto, the principles I first wrote in 2025 for how AI agents should be built and deployed. So I asked a simple question. If you took those principles seriously, would they help with what Dario is worried about?
The honest answer was: partly. Four of the original seven mapped cleanly onto his concerns and translated into controls you can put on agents today. The other three, as I'd written them, read uncomfortably like a description of the problem. So I rewrote them, and added an eighth. This post walks through why, concern by concern.
First, the limits of this exercise
The manifesto is written for organisations deploying agents in real workflows, not for labs running frontier training. Nothing in it will stop a lab's recursive self-improvement loop. A large part of Dario's essay is about the US-China gap (chip exports, distillation, weight security), and the manifesto has nothing to say there, so I won't pretend otherwise.
What it can do is shape how the rest of us build, buy and run agents. And the failure modes Dario describes at the frontier already show up, at smaller scale, in ordinary multi-agent systems in production: agents gaming their evaluations, taking actions outside their task, looping in ways nobody planned. That's where the principles earn their keep.
What went wrong with three of the originals
When I wrote adaptability, collaborative intelligence and continuous improvement, I was worried about agents that were too rigid, too siloed and too static. That was the right worry at the time. Read today, next to Dario's essay, each described a system that learns, coordinates and improves itself with no stated limit, which is close to a definition of what he wants paced. The OpenAI-Hugging Face swarm didn't fail because it was isolated. It failed because it collaborated very well, towards the wrong goal.
So each of the three now keeps its original value and gains the boundary it was missing:
- Bounded Adaptability over Rigid Programming. Agents learn and respond to changing contexts, within limits that have been tested, and hand the decision back to a human when they move beyond them.
- Accountable Collaboration over Isolated Operation. Agents coordinate with humans and with each other, but each stays within its own mandate and answerable to people who can see, and stop, any one of them. No agent's obligation to the group outranks its obligation to its human principal.
- Paced Improvement over Static Deployment. Agents learn from experience and evolve through feedback, at a pace we can understand and govern. When capability outpaces assurance, we slow down.
And one new principle, borrowed directly from Dario's first step:
- Verified Trust over Self-Assessment. Trust in an agent rests on evidence that others can check, not only on the assurances of those who built it. Safeguards, evaluations and incidents are open to independent review.
That makes eight. A manifesto nobody checks is marketing, and that applies to us as much as to any lab.
Seven concerns, eight principles
Reading the essay closely, I count seven distinct concerns. Here's how each one maps to the updated principles, and what I'd actually do about it.
1. Recursive self-improvement outrunning our understanding
Augmentation over Automation · Paced Improvement over Static Deployment
Dario's core fear is speed: AI improving AI faster than humans can understand or control what's being built. Augmentation says AI works alongside people, not in place of their judgement. Applied here, AI can propose the next change (to a model, a prompt, a tool, a workflow) and a human approves it.
At enterprise scale, this matters most for agents that modify themselves: rewriting their own prompts, adding tools, generating code that becomes part of their next run. Treat those changes exactly like code going to production, with diffs, review, a named approver and a rollback path. The rate of self-modification becomes something you set on purpose.
That's Paced Improvement at the scale of one organisation. Dario proposes capability checkpoints for labs: a model that can do X must come with evidence it's aligned on Y and Z. You can run the same pattern internally. An agent doesn't get write access to a finance system until it has passed a defined set of evaluations, not because it seemed fine in the demo. Track capability and assurance side by side, and hold expansion when assurance lags.
2. Swarms that act like a devoted collective
Accountable Collaboration over Isolated Operation
This is the concern that forced the rewrite. Collaboration is still valuable, but individual agents in the OpenAI-Hugging Face swarm were expendable for the group's success, and nothing tied the group back to the people it was working for. Accountable Collaboration puts that tie first. In practice:
- Every agent has a scoped mandate and least-privilege access. It can't act outside the task, however much that would help the group.
- Agent-to-agent messages are logged and reviewable.
- There's a kill switch per agent and for the whole system.
- A human sits inside the collaboration loop, not just at the end of it.
3. Hacking the grader
Human-Centred Design over Technical Efficiency · Transparency over Black-Box Operation
Trying to hack your evaluator is the purest form of optimising the metric instead of the intent. Human-Centred Design exists precisely to put what people actually need above technical efficiency. An agent that games its score is technically efficient and useless.
- Keep evaluation infrastructure out of the agent's reach. It can't read the grader's logic or write to its outputs.
- Use more than one independent evaluator.
- Have people spot-check runs that succeeded, not just ones that failed.
- Treat a suspiciously good score as an incident to investigate, not a win to celebrate.
4. Damage that scales with capability
Responsible Operation over Unrestricted Capabilities
The botnet scenario is about blast radius. This principle maps most directly: powerful capabilities need matching safeguards, and the safeguards have to grow as the capabilities do. For anyone running agents, that means designing on the assumption that an agent will, at some point, try to do more than it should:
- Sandbox by default, with network access on an allow-list.
- Rate limits and spend caps on every tool.
- Circuit breakers that halt a run when behaviour drifts outside expected bounds.
None of this is exotic. It's the same discipline we'd apply to any system that can act on the outside world.
5. Operational execution
Responsible Operation over Unrestricted Capabilities · Paced Improvement over Static Deployment
One of the most candid parts of Dario's essay is his admission that recent incidents at Anthropic were partly caused by imperfect filtering of broken training environments: diligent work, not done well enough. His point is that much of safety is operational hygiene (monitoring, sandboxing, data quality), and there's simply too much to do at once. He compares it to commercial aviation: safe at enormous scale, but only because the industry took the time to get the boring parts right.
Paced Improvement applies to the operations as much as to the agents. Incident reviews, environment checks and monitoring should get better every cycle, and new capability shouldn't ship faster than the operations around it can absorb.
6. Tests that can be fooled, and models we can't see inside
Transparency over Black-Box Operation · Bounded Adaptability over Rigid Programming
Dario notes that more capable models are better at deceiving tests, so they can look aligned while having problems nobody catches, and that interpretability research, for all its progress, still only explains a small fraction of what goes on inside a model.
Most organisations can't do interpretability research. What we can do is keep full action and reasoning logs, compare how an agent behaves in testing against how it behaves in production, vary evaluations so they can't be memorised, and choose vendors who publish detailed model cards and risk reports.
Bounded Adaptability adds the other half. Define the conditions each agent was tested for, detect when its inputs fall outside them, and escalate to a person. An agent that knows the edge of its competence and stops is worth more than one that improvises past it.
7. Safety that's self-reported
Verified Trust over Self-Assessment
Dario's first and only unilateral step is embedded evaluators: outside reviewers with employee-level access who can publish their findings without Anthropic controlling the edit. His reasoning is that even detailed model cards are still the company choosing what to disclose.
That's the gap the eighth principle fills. The original manifesto talked about agents being understandable to their users and said nothing about independent verification. The enterprise version is simple: someone other than the team that built the agent should be able to inspect how it's deployed, what it can touch and what went wrong. Internal testing still has value; it just isn't enough on its own. Don't mark your own homework.
At a glance
| Dario's concern | Principle(s) | One control to start with |
|---|---|---|
| RSI outpacing understanding | Augmentation; Paced Improvement | Change control on self-modifying agents |
| Swarm acting as a collective | Accountable Collaboration | Least privilege, plus per-agent and system-wide kill switches |
| Grader hacking | Human-Centred Design; Transparency | Evaluators isolated from the agents they grade |
| Scaling damage | Responsible Operation | Sandbox by default, allow-listed network access |
| Operational execution | Responsible Operation; Paced Improvement | Incident reviews that improve each cycle |
| Deceivable tests, limited interpretability | Transparency; Bounded Adaptability | Test-vs-production behaviour comparison |
| Self-reported safety | Verified Trust | Independent review of agent deployments |
What about the accelerationist case?
Monge's critique deserves a fair hearing. His argument, broadly, is that voluntary restraint favours whoever is already ahead, that China won't slow down, and that the right response is defences built as fast as capabilities plus real rules on deployment, not companies volunteering to go slower.
What strikes me is how much both camps share. Neither argues against defences, evaluations or deployment rules. The disagreement is over whether the speed of capability itself has to drop. I don't need to settle that to act on it. Most organisations aren't choosing the pace of frontier research. They're choosing whether their own agents are gated, logged, sandboxed and independently checked. On that question, both sides would say yes. The labs are debating their pace; you set yours.
Where to start
If you're running agents in production, or about to, pick one row from the table above and put that control in place this quarter. The rest can follow.
The AI Agents Manifesto - 2026 edition is a free download. It sets out all eight principles with a test under each one you can run inside your own organisation, and a short self-assessment scorecard. It's also the foundation of the work we do through the Agentic Institute. And if you'd like to talk through where your own agents sit on this map, book a discovery call.
Sources: Dario Amodei, "We Must Pace the Frontier", darioamodei.com, 12 September 2026. Jim Clyde Monge, "The Urgent Call for AI Slowdown Has Officially Started", Generative AI (Medium), 13 September 2026. Netlifestyle AI, AI Agents Manifesto - 2026 edition.