Unify: Popular Agents Failed to Learn on Its New Continual ARC Benchmark
OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience. On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.
Context
If widely used agents cannot remember or transfer what they just practiced, teams cannot assume that running an agent longer will make it better.
Daniel Lenton of Unify posted that OpenClaw, Hermes Agent and Prime Agent do not meaningfully learn from experience. On Continual ARC, Unify's new benchmark, he said none of them were able to memorize anything of value and none were able to generalize to new tasks, while training a small 9 billion parameter model's weights did both. Unify also published a benchmark page, a blog post, and a paper describing the test.
Deep dive
Daniel Lenton, posting about work at Unify, said OpenClaw, Hermes Agent and Prime Agent do not meaningfully learn from experience. On Continual ARC, which he called Unify's new benchmark, he said none of those agents were able to memorize anything of value and none were able to generalize to new tasks. He said training a small 9 billion parameter model's weights did both.
Unify's benchmark page describes Continual ARC as a test of skill acquisition over a working lifetime, credited to Daniel Lenton, Yusha Arif and Haris Mahmood. According to that page and Unify's blog, the benchmark serves a long stream of ARC puzzles in which the same tasks return with fresh inputs while new tasks keep arriving. After the stream, the learner is frozen and tested on fresh instances of tasks it already met, which is meant to measure memorization, and on tasks it never met, which is meant to measure generalization.
The blog says a 9 billion parameter model's weights, trained to imitate a stronger model, improved on both the puzzles it had seen and a set of new ones, while note taking harnesses did not show that kind of gain. Those details come from Unify's own writeups, not from the lead post, which named the agents and the headline result but did not list every score.
The result is Unify's own evaluation of its benchmark. Other teams have not yet been cited here as having confirmed the same ranking on a public leaderboard.
What to expect next
Watch for other labs to run the public Continual ARC stream and for Unify to publish verified results on its private held out set, which the company has invited outsiders to submit to.
The open question is whether later agent harnesses can close the gap that Lenton reported, or whether teams will need to train model weights to get lasting memory and transfer. Until independent runs appear, these figures should be read as Unify's results on its own benchmark.