Simran Arora Introduces the Together Compiler for Continual Compilation of AI Inference
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Context
An agent driven compiler that retunes serving as models and traffic change could cut the cost of running popular open models.
Simran Arora (@simran_s_arora) posted from Together AI (@togethercompute). Inference compilers turn model math into fast GPU kernels. OpenRouter is a router across many model APIs.
Deep dive
Simran Arora (@simran_s_arora) posted on Nov 4 introducing the Together Compiler, which she described as continual compilation for AI inference.
According to the post, today's inference is polymorphic: on OpenRouter, 80% of traffic goes to models released in the last three months, and the top ten open models each use a different architecture. Together Compiler uses agents to keep optimizing deployments as those shapes and traffic patterns change.
She listed research highlights the team claims: 252x more optimization experiments per hour than prior agent inference optimizers; state of the art tokens per second on InferenceX workloads tried, including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1 across NVIDIA, AMD and Trainium2; continuous recompilation that serves many trillions of production tokens per day and adapts within a few hours; and design lessons from ThunderKittens and Megakernels.
These figures come from Together and are not independently confirmed in the post.
The post passed about 125,000 views.
What to expect next
Watch for independent throughput tests on the named models and chips, and for Together to open more of the compiler to customers. Rival inference stacks at Fireworks, Groq and the cloud giants may answer with their own continual tuning. If the 252x experiment claim holds, agent written kernels may become a standard part of serving. Together may publish more workload charts so outsiders can reproduce the TPS numbers.