Anthropic's Frontier Red Team says Claude formally verified security properties in over 100 open source libraries, with Theorem
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here:
Context
If AI can write machine checked proofs about real code at this scale, proven security guarantees could move from rare, expert only work toward a routine part of software development.
Formal verification means mathematically proving that code follows a set of rules, and it has been costly because proofs take expert effort and are often far larger than the code. Anthropic's Frontier Red Team worked with Theorem, a company that supplied formal models of the processor and operating system used in the proofs.
Deep dive
On October 22, Anthropic said it used Claude to prove security properties in more than one hundred real world open source libraries over a few weeks, in collaboration with Theorem. All figures below come from Anthropic's own write up, and the write up says to Anthropic's knowledge this is the first large scale effort to apply AI formal verification to production software at the assembly code level.
Anthropic says Claude proved that fuzz test entry points never crash for any input under 64 MB, across 239 entry points in 123 libraries. The proofs cover compiled RISC V binaries, total about 180 million lines in the Rocq proof language, and a typical proof runs to about 600,000 lines. Anthropic says a Rocq checker validates the proofs, so the people who rely on them need to trust the formal models of the processor and Linux, the exact theorem statements, and the checker.
Anthropic says the work also found bugs, including a rediscovery of a known libexpat vulnerability, which the write up identifies by its CVE number and an out of bounds write that appeared only on rare inputs of around 20 MB. It says Claude patched the bugs it found and that it audited the patches for shortcuts.
The write up lists limits. Formally verified does not mean bug free, the proofs assume a single threaded static binary on a RISC V processor under Linux, proofs can take hundreds of millions of tokens, and Anthropic has not published every proof. It says it is releasing a subset of the proofs and the models so others can check them.
What to expect next
Watch for independent checks of the published proofs and for security researchers to test whether the stated assumptions and trusted components hold up. Anthropic says the main gap is extending the models beyond single threaded static RISC V binaries to the x86 and ARM systems most people run. Cost is the other open question, since proofs took very large amounts of compute. Theorem says it will run the same pipeline for other open source projects, which would show whether results carry over.