A new benchmark of 50 unpublished theoretical physics problems finds GPT 6 Astra in Codex solving 98% of them
GPT-6 Astra got +80% on a Physics benchmark. It insisted the expert results were wrong and its calculations were right. A team checked. And yes. GPT-6 Astra was right. New score: 98%
Context
Research level physics calculations are a hard test of whether AI agents can do real science, and the paper also finds signs that leading models have memorized much of the literature.
This item is secondhand: Freddy Vega is summarizing a post by physicist Pratik Rath, one of the paper's authors. The claim that the model was right where expert answers were wrong comes from that framing, while the 98% figure is in the paper itself. The paper was posted to arXiv on October 9.
Deep dive
On October 14, Freddy Vega posted that GPT 6 Astra got over 80% on a physics benchmark, insisted the expert results were wrong and its own calculations right, and that after a team checked, its new score was 98%. He was quoting Pratik Rath, a coauthor of the paper.
The paper, TheoPhysEval, has 37 authors coordinated by Praneeth Netrapalli of Google DeepMind and Suvrat Raju of ICTS. It presents 50 unpublished research level theoretical physics problems with checkable answers, each of which the authors say a world expert could solve in about a day or less. Agents were tested without human hints.
With web search and the ability to write scripts, GPT 6 Astra running in Codex reached 98% at an average cost under $3 per problem, according to the abstract. A summary of the paper lists Claude Opus 5.5 in Claude Code at 46 of 50 and Gemini 3.8 Flash in Antigravity at 34. In an isolated track with no internet, Astra and Opus both scored 92%.
The authors write that small accuracy changes without web search suggest significant memorization of the literature, and they discuss why agents have not yet produced breakthroughs in theoretical physics despite strong problem solving.
What to expect next
Watch whether other groups reproduce the results, and whether the disputed answers that the model reportedly got right are documented in later versions of the paper.
Also watch for follow up work testing whether agents can move from solving set calculations to producing new results in theoretical physics, which the authors say has not happened yet, and whether labs start citing TheoPhysEval in model launches.