OpenAI Developers says ChatGPT's storage now serves data in milliseconds to more than 1.2 billion users across 40 regions, with a 12x cut in global median read latency on Azure Cosmos DB
We serve data in milliseconds to more than 1.2 billion users across 40 geographic regions. To reach that scale, we worked closely with @AzureCosmosDB on our storage architecture, achieving a 12x improvement in global median read latency.
Context
It is a rare look at the plumbing behind a service used by over a billion people. Faster reads make ChatGPT feel quicker worldwide, though all figures come from OpenAI.
OpenAI has published a two part engineering series on scaling online storage for ChatGPT, built with Microsoft's Azure Cosmos DB. Coverage of the second part says the platform is called Habitat and that median read latency fell from about 120 ms to about 10 ms. The post was published at 1:23 PM Eastern on October 23.
Deep dive
OpenAI Developers posted at 1:23 PM Eastern on October 23 that OpenAI serves data in milliseconds to more than 1.2 billion users across 40 geographic regions, and that working with Azure Cosmos DB on its storage architecture brought a 12x improvement in global median read latency. These are OpenAI's own figures.
The post points to OpenAI's engineering write up on scaling online storage for ChatGPT, published in two parts. Summaries of the second part say the storage platform is called Habitat and that median read latency dropped from roughly 120 milliseconds to about 10 milliseconds. They describe three main moves: replicating critical data close to users, adding a cache inside each region that is cleared through Cosmos DB's change feed, and rewriting a service from Python to Rust.
The user count is notable on its own: 1.2 billion users across 40 regions frames the scale its storage has to handle. The post is an engineering update rather than a new product that shows how OpenAI runs a core part of ChatGPT on Microsoft's cloud.
What to expect next
Engineering posts like this tend to be followed by talks and deeper write ups, so watch for OpenAI or Microsoft sharing more on Habitat's design and costs. The figures will also be a reference point as OpenAI spreads its infrastructure across more providers, since storage is one of the harder parts to move. Any outage or latency complaint would test how the regional design holds up.