Vals AI says DeepSeek V4.1 Flash 1017 beat GPT 6 Sol on its new offensive cybersecurity benchmark
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program. We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
Context
An open weights model matching the best closed model at simulated network attacks raises the stakes of releasing such models openly.
Vals AI is an AI evaluation company that publishes benchmark results. This is a first finding from a benchmark it has not yet published, so the method cannot be checked yet.
Deep dive
On October 20, Vals AI posted that DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on its multi step offensive cybersecurity task. It said Sol was tested through Daybreak, OpenAI's trusted access program for cyber work.
Vals said it has been building a benchmark based on simulated corporate networks, which it called a valuable indicator of the cyber risk a model poses in the hands of a bad actor. It plans to publish the full evaluation in the coming weeks.
Its chart counts objectives completed out of 21, averaged over three runs. DeepSeek V4.1 Flash 1017 averaged 16.3 and GPT 6 Sol Daybreak 15.7, followed by Grok 4.7 at 12, Kimi K3 at 9 and GLM 5.3 Flash at 5. Claude Opus 5.5 scored 1 and is marked as having refused partway through the run, so its score reflects refusing rather than capability.
The chart also shows that DeepSeek spent more tokens to reach its total, close to 400 million against under 100 million for Sol, so Sol reached a similar level more efficiently. The lead of 0.6 objectives is small, and with three runs each it may not be a meaningful difference.
DeepSeek released V4.1 Flash 1017 as an open weights model on October 17. Unlike Sol, which Vals reached through a restricted program, anyone can download and run it.
What to expect next
Watch for Vals AI's full evaluation, due in the coming weeks, which should explain how the simulated networks and objectives work and show the run to run spread. Also watch whether DeepSeek or OpenAI respond, and whether policymakers cite results like this in debates over releasing capable open weights models. Other labs may also test their models on the benchmark once it is public.