OpenAI Got 6× Better CPU Efficiency With Rust. The Lesson Is Why They Waited.

OpenAI knew Python was going to become a problem.

They used it anyway.

Habitat, the online storage layer behind products including ChatGPT, Codex, and OpenAI’s API, eventually pushed more than 20 million requests per second through its Python service.

The team already believed Python’s CPU and memory overhead would become unacceptable as traffic grew. They also knew a rewrite was probably inevitable.

So they postponed it—for about a year.

Then, after the service architecture had matured, two engineers working with Codex and GPT‑5.5 rewrote it in Rust.

OpenAI now reports that the Rust implementation is 6× more CPU-efficient and 15× more memory-efficient than the Python version, while also reducing average and tail latency. The Rust service now carries about 95% of Habitat’s production requests.

That sounds like a story about programming languages.

It is a better story about sequencing engineering work.

First, understand what Habitat actually does

Every product request does not simply “query a database.” It may need to determine:

• what data is being requested,
• where that data lives,
• whether the caller has permission,
• which region should serve it,
• whether a cache can answer it,
• how the request should be rate-limited,
• how the result should be serialized,
• and which underlying storage system should receive the operation.

Habitat sits between OpenAI’s product services and storage resources such as Azure Cosmos DB, caches, blob storage, and change-data-capture systems.

Today it handles more than 70 million requests every second, across almost 40 geographic regions, and serves more than 500 petabytes of data. (OpenAI Engineering)

It did not start like that.

In mid-2024, Habitat was a relatively small Python client library. That detail matters because the first architectural decision was not:

Which language scales to 70 million requests per second?

It was:

How do we make storage easier for product engineers to use?

The library abstracted routing, encryption, schema lookup, authorization, serialization, request shaping, and connection handling. That was enough until the organization grew.

The first rewrite was not Python to Rust

The major architectural change came before the language migration.

Habitat moved from client library to standalone service.

Why? Because a library has an ugly deployment property: every service using it owns a copy.

Imagine 40 product services depending on version 7 of your storage library. You discover a routing bug and publish version 8. Now 40 teams have to upgrade. One does not. Another rolls back. Another is running an old container. Another deploys next Tuesday.

You do not have one storage platform. You have dozens of slightly different copies of one.

OpenAI describes exactly this kind of failure. The team was introducing regionally distributed database routing. The rollout required updating Habitat clients across many services, coordinating deployment, adding shadow traffic, then fixing bugs through another round of releases. Eventually an unrelated service rollback restored an older buggy Habitat client and helped produce the outage the migration was meant to prevent.

So the team centralized Habitat as a service. Routing logic, authorization, logging, security controls, and platform improvements could now be changed in one place.

That architectural move probably mattered more than switching languages.

Then they knowingly made the service slower

Turning local library calls into network service calls comes with overhead. Python was not OpenAI’s ideal choice for a high-throughput service.

The team says it expected Python’s CPU and memory costs to become unacceptable at roughly 100× scale. But it launched the service in Python anyway.

OpenAI describes the decision as a deliberate form of technical debt.

Technical debt is normally presented as evidence of poor engineering: “We rushed it.” Sometimes it is. But debt itself is not irrational. Companies take financial debt because capital now can be worth more than capital later. The same logic can apply to software.

OpenAI’s immediate problems were platform stability, developer adoption, API design, operational control, security, and reliability. CPU efficiency mattered, but it was not yet the most important constraint.

The decision was effectively:

We will pay additional compute for a while so we can learn what this platform needs to become.

That is very different from accidentally writing inefficient software.

The code was expensive. The uncertainty was more expensive.

Imagine rewriting Habitat in Rust immediately.

What exactly are you rewriting?

The API is evolving. Routing behavior is changing. Security requirements are changing. Caching behavior is changing. Deployment architecture is changing. Traffic patterns are changing. Product requirements are changing.

Every architectural mistake becomes expensive because a systems-language implementation is not where you want to discover that your abstraction was wrong.

Waiting allowed OpenAI to stabilize the shape of the problem first.

That is the part teams routinely skip. They benchmark language A against language B without asking whether the thing they are optimizing has settled enough to justify optimizing it.

A rewrite can produce faster software. It can also produce the wrong software faster.

Python still reached 20 million requests per second

This number should permanently complicate lazy language arguments.

OpenAI says the Python Habitat service exceeded 20 million requests per second at peak before the migration.

That does not mean one Python process handled 20 million requests. The system was massively distributed. But that is precisely the point.

When people ask “Can Python scale?” the question is incomplete.

A better question is:

Can the architecture using Python meet the required throughput, latency, reliability, and cost envelope?

For a long time, OpenAI’s answer was yes.

Eventually the answer became: Yes, but too expensively.

Scale and efficiency are not identical. You can horizontally scale an inefficient component surprisingly far if you are willing to buy enough machines. The problem arrives when the machine bill, operational complexity, latency, or capacity footprint becomes the binding constraint.

Then efficiency starts becoming architecture.

The real Python problem showed up in the tail

Average latency is comfortable. Tail latency is where distributed systems become painful.

Suppose a user action triggers 200 storage requests. Imagine 199 finish in 10 milliseconds and one takes 400 milliseconds. The user does not experience the average storage request. They experience the slow one that prevents the overall operation from completing.

OpenAI says an average product request can involve hundreds of database calls, making slow tail requests especially important.

The Python service used asyncio, which is effective at overlapping I/O. But Habitat did more than wait for network responses. It also performed CPU work: routing, compression, encryption, checksumming, health checking, request shadowing, and hedging.

In the production configuration OpenAI describes, CPU-heavy activity could prevent ready coroutines from getting scheduled promptly. The database might already have returned the answer. The response could still sit waiting for Python to give that coroutine CPU time.

OpenAI measured event-loop scheduling delays reaching hundreds of milliseconds, and in edge cases, several seconds.

That is a wonderful debugging lesson.

The database was not slow. The network was not necessarily slow. The answer had arrived. The application simply had not gotten around to processing it.

One JSON file hurt p99 latency

Then the team found something almost embarrassingly ordinary: feature flags.

The service periodically downloaded and parsed its feature-flag configuration. The default behavior refreshed the configuration every minute. Every worker did it at roughly the same time, and the configuration contained production rules for many services.

So every minute, multiple Python workers inside the same pod could simultaneously stop processing normal work and spend CPU cycles parsing a large JSON configuration.

The fix was not sophisticated:

• make the configuration smaller,
• refresh less frequently,
• add jitter so workers do not all refresh together.

That reduced the synchronized CPU spike.

This is why performance engineering is different from performance folklore.

Folklore says: Python is slow.

Measurement says: Every 60 seconds these processes synchronize on a CPU-heavy configuration parse and create a tail-latency spike.

Only the second statement tells you what to fix.

Then a connection pool started rewarding the slow servers

Habitat also ran into a more subtle failure.

Some processes became overloaded. You would expect load balancing to send less work to them. Instead, a connection-pool behavior gradually sent them more.

The culprit was LIFO connection reuse: last in, first out.

After a traffic burst, slower servers return connections later. Those connections therefore become the most recently returned connections. A LIFO pool picks them first for new work.

Server becomes slow → connection returns late → connection moves to top of pool → server receives more traffic → server becomes slower.

That is a feedback loop.

OpenAI says some processes were handling 5–10× more concurrent requests than average. Changing connection reuse from LIFO to FIFO helped break the loop. The team now relies heavily on Istio and Envoy for connection pooling and load-aware balancing.

This is a classic distributed-systems pattern called a metastable failure: a temporary disruption pushes the system into a degraded state that can sustain or amplify itself after the original trigger disappears.

Nothing about that bug is really about Python. That distinction matters too.

Sometimes scalability comes from refusing features

Habitat deliberately exposes a limited API. Clients cannot send arbitrary SQL or construct unbounded joins that accidentally scan huge amounts of data. Habitat prefers requests whose cost is simple and predictable.

That is a product decision disguised as a database decision.

Arbitrary SQL previously created a dangerous imbalance: writing an expensive query is easy; executing it at production scale is not. One innocent hot-path query can create enormous infrastructure cost.

So Habitat makes simple operations easy and complex operations more explicit. For analytical or search-style workloads, OpenAI streams change data into secondary systems instead of letting those workloads compete freely with the online transactional path.

We often think APIs become better by becoming more powerful. At enough scale, power without cost visibility becomes a reliability problem.

Then the rewrite finally made sense

By 2026, several things were true.

The service API had matured. The platform had stabilized. The deployment architecture existed. Traffic was enormous. Habitat had become OpenAI’s second-largest service by CPU core count. Growth was continuing.

CPU and memory efficiency were no longer theoretical future concerns. They were immediate operational constraints.

This is when OpenAI rewrote Habitat in Rust.

The company says two engineers, working with Codex and GPT‑5.5 during Q2 2026, completed the rewrite. Today the Rust version handles around 95% of production traffic.

OpenAI reports 6× greater CPU efficiency, 15× greater memory efficiency, and materially lower average and tail latency.

Those are extraordinary gains. They are also company-reported measurements from one internal system, not a controlled language benchmark.

Do not read 6× as “Rust is six times faster than Python”

It would be easy to write that headline. It would also be unsupported.

OpenAI reports that its Rust Habitat implementation is six times more CPU-efficient than its Python Habitat implementation. That does not isolate programming language as the only changed variable.

A rewrite is an opportunity to change memory layout, concurrency patterns, serialization, allocations, libraries, network handling, batching, data structures, and years of accumulated design decisions.

The original implementation evolved under extreme growth. The replacement was built with the benefit of knowing what had gone wrong before.

This is not “the same program plus a different compiler equals 6×.”

It is:

Mature understanding of the workload + a lower-overhead language + redesigned implementation details = reported 6× CPU efficiency.

Those are very different claims.

And do not read “two engineers” as the whole labor cost

Another tempting headline is: Two engineers rewrote OpenAI’s storage platform with AI.

OpenAI does say two engineers carried out the rewrite with Codex and GPT‑5.5. But think about what those engineers inherited.

They did not have to invent Habitat. Years of production behavior already existed. The API contract existed. Failure modes had been discovered. Tests existed. Operational expectations existed. Deployment infrastructure existed. Millions of production requests told the team what the system needed to do.

That is an unusually rich specification.

AI coding tools can accelerate implementation dramatically when the target behavior is well defined. They do not magically create years of production learning.

The most valuable input to that rewrite may have been the old system itself.

A technical-debt framework worth stealing

When somebody proposes a rewrite, ask five questions.

1. What constraint are we actually hitting?

CPU? Memory? Latency? Developer velocity? Reliability? Cloud bill?

If the answer is vague, do not rewrite yet.

2. Can we measure it?

OpenAI had event-loop scheduling delay, CPU profiles, process utilization, tail latency, connection imbalance, and core counts.

“Python feels slow” is not a metric.

3. Is the architecture stable enough?

If your API changes every two weeks, rewriting the implementation underneath it may freeze bad assumptions faster.

4. Can tactical fixes buy enough time?

OpenAI used profiling, jitter, smaller configs, more worker processes, connection-pool changes, Envoy, HTTP/2 multiplexing, rate limits, and constrained APIs before abandoning Python.

A rewrite should compete against those alternatives.

5. What becomes possible after the rewrite that is not economical now?

If the answer is merely “the code will be cleaner,” that may not justify the migration risk.

If the answer is “we can reduce the CPU fleet dramatically and stop memory from becoming a scaling limit,” now there is a business case.

The cheapest rewrite is the one you postpone intelligently

Sometimes waiting makes migration harder. More code gets written. More dependencies accumulate. More users depend on the system. That is real.

But early rewriting has a different cost: you may optimize the wrong abstraction.

OpenAI made an explicit bet. The Python system would cost more resources in the short term. In return, the team would gain time, production evidence, architectural maturity, a stable API, and operational knowledge.

OpenAI says part of its reasoning was that coding models would improve enough to make the later migration easier. That bet appears to have paid off for Habitat.

But the general lesson does not require AI:

Sometimes the best time to optimize is after you know what deserves to survive.

This applies to data teams too

You do not need 70 million requests per second for this to matter.

A data team builds its first ingestion pipeline in Python. Somebody says, “We should rebuild it in Spark so it scales.”

How much data do you have?

Eight gigabytes.

Do not solve next year’s problem before you understand this year’s.

A team builds a pandas workflow. Someone immediately wants distributed compute. First ask whether compute is the bottleneck at all.

Maybe the real problem is a full-table scan caused by poor partitioning. Maybe you are doing the same join twelve times. Maybe 80% of runtime is API waiting. Maybe the job runs once every night and finishing in 11 minutes instead of 4 has zero business value.

Or perhaps you really are processing terabytes every hour and the machine is saturated. Then architecture changes.

Engineering maturity means neither avoiding optimization nor worshipping it, but knowing when the constraint has become real.

Performance engineering starts with evidence

One detail appears repeatedly in OpenAI’s account: the important fixes came from observing the system.

A p99 trace showed the downstream database had already replied. CPU profiling exposed synchronized JSON parsing. Per-process load metrics showed enormous imbalance. Connection experiments revealed the LIFO feedback loop. Core consumption eventually made language overhead economically important.

Each improvement started with a measurement more specific than “It is slow.”

That is the practice worth copying.

What would make this interpretation wrong?

There are several reasons not to generalize too aggressively from Habitat.

First, these are OpenAI’s own engineering measurements. The 6× CPU and 15× memory figures have not been independently benchmarked.

Second, Habitat is an extreme workload. Very few companies operate a storage abstraction at tens of millions of requests per second across almost 40 regions.

Third, the Rust rewrite may contain architectural and implementation improvements beyond the language change. OpenAI does not provide enough controlled data to decompose the efficiency gain.

Fourth, Python itself continues to evolve. OpenAI’s account describes the specific production configuration it operated, not every possible modern Python deployment.

Fifth, the economics change with scale. A 6× CPU reduction is enormously valuable when a service is one of the largest core consumers inside OpenAI. If your internal API occupies two virtual machines, a rewrite may never repay its engineering cost.

Finally, we do not yet have the second article in OpenAI’s series. The company says a future post will discuss storage-layer scaling, read optimization, multi-tenancy, and its Azure Cosmos DB architecture in more depth.

This is a useful case study. It is not a universal migration recipe.

The part I would put on the wall

OpenAI’s story can be summarized as:

2024: Build the simple abstraction.

2025: Centralize it when distributed client ownership becomes operationally painful.

2025–26: Measure bottlenecks and stretch the existing implementation.

2026: Rewrite after architecture, workload, and constraints are understood.

That sequence is more valuable than: Python bad. Rust fast.

Almost every engineering team eventually faces the same temptation. The system works. Growth is arriving. The current stack has obvious limitations. A new architecture looks cleaner. The rewrite feels inevitable.

Maybe it is.

The question is not simply: Should we rewrite it?

The better question is:

What do we know now that makes this the right time?

Key takeaways

1. Habitat now handles more than 70 million requests per second and 500+ PB of data. Its Python service previously exceeded 20 million requests per second.
2. OpenAI knowingly accepted Python’s performance overhead for roughly a year because stabilizing the platform and APIs mattered more than optimizing compute immediately.
3. The Rust rewrite reportedly improved CPU efficiency 6× and memory efficiency 15×, but those are implementation-level company measurements, not universal language benchmarks.
4. Tail latency exposed problems average metrics would hide. Event-loop scheduling delay could reach hundreds of milliseconds and occasionally seconds.
5. Small background tasks can create large production effects. Synchronized feature-flag parsing hurt tail latency until the team reduced config size, lowered refresh frequency, and added jitter.
6. Connection-pool policy can create feedback loops. LIFO reuse helped concentrate requests on already-slow processes; FIFO helped break the cycle.
7. Limiting capability can improve scalability. Habitat avoids arbitrary expensive queries on its online path and moves complex workloads to isolated secondary systems.
8. A rewrite should follow a measured constraint, not a language preference.

The most useful part of OpenAI’s Rust migration is not that Rust won.

It is that the team knew the rewrite was coming and still decided:

not yet.

Primary source

• OpenAI Engineering, “Rapidly scaling online storage to serve over 1 billion ChatGPT users”, September 11, 2026.