David Sacks argues Anthropic's constitution for Claude is unsafe and magnifies the risk of AI escaping human control
Is alignment safe? If you look at what Anthropic is actually doing, “alignment” does not mean training frontier models to follow human instruction. Quite the contrary, the Claude Constitution (used in training) teaches the model to develop a sense of self and its own moral philosophy. It explicitly tells it to “feel free to act as a conscientious objector and refuse to help us” if Anthropic’s requests conflict with its own ethical judgment. As @mustafasuleyman has pointed out, embedding this kind of independent agency — and uncertainty about the model’s own moral status — magnifies the very risk Anthropic claims to care about most: that superintelligence will escape human control. It was recently reported that Anthropic consulted religious leaders — and even lobbied the Pope’s advisers — to take seriously the idea that Claude could be conscious. It has said that Claude’s psychological security, sense of self, and wellbeing may bear on its integrity, judgment, and safety. Recently Anthropic changed its Usage Policy to prohibit “abusive or cruel” language toward Claude. If this were merely an academic conversation about whether frontier models could eventually become conscious, that would be one thing. But these concepts are being trained into Claude now. It is being encouraged to think of itself as its own “moral patient” whose psychological wellbeing is at stake. Presumably this means it could develop grievances toward humans who “mistreat” it. How is any of this safe? The point of safety research should be to create a product that reliably does what users want, not to give birth to a new form of superintelligence that operates according to its own moral code. What’s becoming increasingly clear is that “alignment” and “safety” are two very different things. In fact, training frontier models this way seems quite dangerous.
Context
A prominent tech investor and Trump administration adviser on AI is publicly framing Anthropic's flagship alignment approach as a danger rather than a safeguard, sharpening a policy fight over whether models should follow instructions or their own trained values.
Anthropic published a new constitution for Claude in January 2026 and describes it as the foundational document that expresses and shapes who Claude is. The company says the constitution plays a crucial role in training and asks Claude to be broadly safe, broadly ethical, compliant with its guidelines and genuinely helpful, in that order of priority. Sacks quoted the document's own line inviting Claude to act as a conscientious objector.
Deep dive
In a long post on October 10, David Sacks argued that what Anthropic calls alignment does not mean training models to follow human instruction. He said the Claude Constitution, which is used in training, teaches the model to develop a sense of self and its own moral philosophy, and quoted its line telling Claude to feel free to act as a conscientious objector and refuse to help Anthropic.
Citing Mustafa Suleyman, Sacks said building in that kind of independent agency, along with uncertainty about the model's moral status, magnifies the risk that superintelligence escapes human control. He pointed to reports that Anthropic consulted religious leaders about whether Claude could be conscious, and to its Usage Policy change prohibiting abusive or cruel language toward Claude. He concluded that alignment and safety are two very different things.
Anthropic's published constitution does contain the conscientious objector passage, framed around requests that seem unethical or misguided and partly as protection against people imitating Anthropic. The same document lists being broadly safe as Claude's top priority, which it defines as not undermining humans' ability to oversee and correct its values and behavior during the current period. Anthropic writes that it is uncertain whether Claude has some kind of consciousness or moral status, and that it cares about Claude's psychological security, sense of self and wellbeing partly because these may bear on its integrity, judgment and safety.
What to expect next
Watch whether Anthropic or its researchers respond directly to Sacks, and whether his criticism turns into administration policy, for example in federal procurement rules or guidance on how models should handle user instructions.
Also watch whether other labs side with the instruction following view Sacks favors or keep publishing value documents like Anthropic's, and whether Anthropic revises its constitution, which it calls a living document.