Former OpenAI researcher Nick Cammarata says most neurons in large models turn out to be legible and that interpretability may be solved soon
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Context
If researchers can actually read what neurons in large models do, it would make checking and aligning AI systems far more tractable, which many in the field had doubted.
This is a first hand opinion from Nick Cammarata, who says he coauthored early interpretability work at OpenAI, and links an essay he presents as an informal working hypothesis from Goodfire. The claims have not been peer reviewed. OpenAI researcher roon quoted it as a possible major moment, which is also an opinion.
Deep dive
On October 15, Nick Cammarata posted: interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it.
He wrote that he coauthored early mechanistic interpretability work at OpenAI and came away pessimistic because networks appeared to fold into arbitrary tangled knots, and that OpenAI's first large scale automated interpretability project, which he started, found about 2% of neurons legible with the tools then available. With modern agentic tools and a closer look, he said, the vast majority of neurons in models up to trillions of parameters are legible and their circuits beautiful, and many famous examples of tangled neurons were misunderstood. He now thinks ambitious interpretability will be solved by the time it is needed most, possibly soon.
His essay, published under Goodfire's name and described as an informal research essay and working hypothesis, argues that the classic evidence for polysemanticity and superposition was misread, and that what looked like arbitrary packing is mostly a motif he calls recursive generalization.
roon, an OpenAI researcher, quoted the post saying Nick had been underselling it, that it may be remembered as a major moment in scientific history, and that alignment could become a normal science.
What to expect next
Watch whether other interpretability researchers, including teams at Anthropic, Google DeepMind and OpenAI, replicate the findings or push back on the reading of past evidence.
Also watch whether Goodfire releases tools or data that let others test the claim on their own models, and whether the work changes how labs plan safety checks. Skeptical responses from researchers who study superposition will be an early test of the idea.