Benchmark Audit Finds Pi Harness Prematurely Terminated 20 Percent of Gemini 4 Argon Runs, With Antigravity Rerun Placing Argon Just Behind Claude Opus 5.5
Gemini 4 Argon is underrated 😱. When auditing the traces, we found that Pi Harness just ends the run whenever it hits a fallback (e.g. for security task) in 20% of the runs. We rerun Argon with Antigravity Harness, and now it beats Astra and sits right behind Opus 5.5. Congrats to the team @GoogleDeepMind and looking forward to the new ckpt.
Context
Agentic coding and research benchmarks depend heavily on execution harness logic, meaning a single fallback handling bug can mask a frontier model's true capability and distort public leaderboard rankings.
FrontierCS 2 is a computer science research and development benchmark introduced in October 2026 where AI agents receive paper codebases with core algorithmic contributions removed and must outperform the original methods without internet or paper access. Initial leaderboard runs evaluated frontier models across standardized execution harnesses including the Pi Harness and the Antigravity Harness.
Deep dive
Researcher Wenhao Chai reported late on October 25 that an audit of execution traces on the FrontierCS 2 computer science benchmark uncovered a major harness bug affecting Google DeepMind's Gemini 4 Argon. When evaluated under the Pi Harness, roughly 20 percent of Gemini 4 Argon's evaluation runs terminated prematurely whenever the model triggered a fallback condition, such as during security sensitive computer science tasks.
To isolate the model's underlying reasoning and coding performance from the harness failure, the benchmark team re evaluated Gemini 4 Argon using the Antigravity Harness. With the premature termination bug removed, Gemini 4 Argon's score jumped past OpenAI's GPT 6 Astra and landed immediately behind Anthropic's Claude Opus 5.5 near the top of the leaderboard.
In follow up replies, Chai noted that after switching harnesses, Gemini 4 Argon achieved both cost efficiency and task performance on par with Claude Opus 5.5. The team also audited execution traces for the other evaluated models and harnesses and confirmed that the fallback termination bug specifically penalized the combination of Gemini 4 Argon and the Pi Harness rather than other entries.
What to expect next
As AI benchmarks shift from static multiple choice questions to multi hour agentic research loops, harness engineering and error recovery logic increasingly determine where frontier models land on public leaderboards. Benchmark maintainers will need mandatory trace audits and multi harness validation before publishing headline rankings so scaffold incompatibilities are not misread as underlying model capability deficits by developers and enterprise buyers.