CellType Introduces BioSimBench, Built on NVIDIA AI, and Finds Frontier Models Predict Only Part of How Cells Respond
Can LLMs simulate biology? We tested frontier models on real lab experiments: predict how cells respond to drugs, gene knockdowns, and immune signals. Even the best get only part of the response right. Lots of room to improve. Introducing BioSimBench, built on @NVIDIAAI. 馃У
Context
A benchmark built on real lab experiments shows how far AI still is from simulating biology, a key goal for drug discovery.
CellType (@celltypeinc) is a biology AI company. The benchmark asks models to predict how cells respond to drugs, gene knockdowns and immune signals, using results from real experiments. It was built on NVIDIA AI tools.
Deep dive
CellType (@celltypeinc) posted on Nov 4 asking whether LLMs can simulate biology, and introduced BioSimBench, built on NVIDIA AI.
According to CellType, the team tested frontier models on real lab experiments, asking them to predict how cells respond to drugs, gene knockdowns and immune signals. CellType says even the best models get only part of the response right, and that there is lots of room to improve.
The post opens a thread with more details, but the main post does not name which models were tested or give their scores. Those results should be read in the linked thread before quoting numbers.
The benchmark matters because predicting cell behavior is one of the main hopes for AI in drug discovery. If models could reliably simulate what a drug does to a cell, labs could skip many costly experiments. A benchmark grounded in measured results, rather than textbook questions, gives a harder and more honest test of that hope. CellType's own finding is that today's frontier models are not there yet.
The post passed 660,000 views.
What to expect next
Watch for model makers to report BioSimBench scores in new releases, and for other labs to test their biology models on it. CellType may publish the full method and data so results can be reproduced. Scores rising quickly on this kind of test would signal real progress for AI in drug discovery, while slow gains would temper claims about AI replacing lab work soon.