Simular says its Agent S4 evolved new interfaces that let an unchanged open Qwen3.8 27B model complete 21.3% of OSWorld V2 tasks, above its reported results for GPT 5.5 and Claude Opus 4.8, at $1.10 per task
Our agent scientist, Agent S4, just got an open 27B model to beat GPT-5.5 and Claude Opus 4.8 on OSWorld V2, with the model's weights left exactly as they were. With the interfaces S4 evolved, Qwen3.8-27B completes 21.3% of the 108 tasks, the highest binary success of any open-source computer-use agent and 3.8x the same model with fixed interfaces. It costs $1.10 per task, against $25.50 for GPT-5.5 and $72.40 for Opus 4.8 馃捀 Full write-up:
Context
If it holds up, better agent tooling alone, not a bigger model, lifted a small open model past older closed ones on computer use, but every number is Simular's own.
Simular is a startup building autonomous computer agents. Its write up, dated October 21, describes Agent S4 as a system that studies a fixed task agent and rewrites the repository of interfaces it uses, and says a paper titled Evolving Agent Interfaces is on arXiv and the code is open source. The X post was published at 12:13 PM Eastern on October 23.
Deep dive
Simular says Agent S4 runs two agents: a task agent that attempts tasks with a repository of interfaces, and a meta agent that reads its trajectories and rewrites that repository through mutation and crossover across a population. The model and harness stay fixed. Simular says the meta agent learns only from synthetic training tasks without success labels and never sees validation or benchmark tasks.
In Simular's table for the 108 task OSWorld V2 benchmark, Qwen3.8 27B with an evolved repository reaches 21.30% binary success at $1.10 per task, against 5.56% for the same model with fixed interfaces, which is the 3.8 times figure. Claude Opus 4.8 in a batched setting is listed at 20.60% and $72.40 per task, and GPT 5.5 at 13.00% and $25.50. On partial credit, though, the evolved setup scores 46.90%, below Opus 4.8 at 54.80% and GPT 5.5 at 49.50%. A variant with a scratch virtual machine scored higher on partial credit, 51.06%, but lower on binary success, 19.44%.
Simular says evolution added 4 new interfaces and 127 extensions to existing ones in the main run, and that on ScienceBoard the evolved repository solved 99 of 169 scientific software tasks against 93 for the seed. The closed models compared are not the newest ones, all results are self reported, and no independent replication was found for this entry.
What to expect next
Simular says the code is open source in its Agent S repository and the paper is on arXiv, so outside groups can test whether the evolved interfaces carry over to other setups. Watch for independent OSWorld V2 runs, comparisons with newer closed models such as Claude Opus 5.5 or GPT 6 Astra, and whether the approach helps larger models as much as Simular suggests it can.