OpenAI Serves LLM Inference at Scale

The scaling challenge behind LLM inference

OpenAI’s public engineering posts repeatedly point out that serving large language models to millions of users is fundamentally different from traditional web‑service workloads. A single inference request must traverse a model with billions of parameters, consume high memory bandwidth, and return token streams within sub‑low latency latency budgets. At the same time, the platform must keep per‑token cost under control and support rapid model updates without downtime. The combination of massive compute demand, strict latency expectations, and the need for global availability makes LLM inference a hard problem that forces trade‑offs across hardware, networking, and orchestration layers.

The post does not publish exact traffic volumes or cost figures; the difficulty is described qualitatively in OpenAI’s blog and engineering notes.

Why LLM inference at scale is a hard problem

LLMs require billions of parameters, high memory bandwidth, and low‑latency compute. Scaling inference to millions of concurrent users forces trade‑offs between latency, cost‑per‑token, and model freshness. Traditional CPU‑oriented serving stacks cannot meet the sub‑low latency latency targets while keeping power consumption reasonable.

How OpenAI serves LLM inference at scale: a 60‑second overview

A user’s request hits OpenAI’s Anycast DNS, which resolves to the nearest edge router. The edge performs TLS termination, request authentication, and sharding based on model version. Requests are forwarded over a high‑throughput private fabric to a regional accelerator cluster (Jalapeño pods). The accelerator runs the model, streams token outputs back through the same path, and logs usage for billing. If the accelerator is unavailable, traffic falls back to a CPU‑based fallback pool.

Global edge topology and traffic routing

OpenAI uses Anycast BGP to advertise the same IP prefix from dozens of PoPs worldwide. Edge routers run a custom L7 load balancer that routes based on model version, token‑budget, and real‑time load. This design ensures users are served from the network-nearest accelerator cluster while keeping the control plane simple.

Failure domains and blast radius

Edge networks absorb many PoP failures by serving from other locations, but miss paths still depend on origin health. Remaining risk concentrates in:

• Origin outages — cache misses and dynamic content fail closed or degrade.
• Misconfigured TTLs / cache keys — personalization bugs and stampedes.
• Edge compute errors — edge logic can fail before origin helps.
• Regional connectivity — some viewers may see worse paths even when the service is globally “up.”

Prefer vendor docs wording over invented PoP counts or latency SLAs.

Request path: from API call to token generation

1. API client sends HTTPS POST to api.openai.com.
2. Anycast DNS resolves to the nearest edge.
3. Edge terminates TLS, validates the API key, and extracts the requested model version.
4. The request is sharded and forwarded over the private fabric to the appropriate accelerator pod.
5. The accelerator runs the inference, streaming tokens back to the edge.
6. The edge re‑encrypts and returns the token stream to the client, while also logging usage for billing.

The sections that follow will dive deeper into each layer of this architecture, examine the mechanisms that make it work at scale, and discuss the trade‑offs OpenAI has chosen.

Key takeaways

• Anycast routing + edge sharding lets OpenAI route a request to the network‑routing-nearest edge node, keeping TLS termination and auth off‑load close to the client.
• Accelerator‑first execution (Jalapeño pods) provides the bulk of inference throughput; a CPU fallback preserves availability when GPUs are saturated or unavailable.
• Model‑version sharding isolates hot models, reduces cross‑tenant interference, and enables independent scaling of each shard.
• Stateless edge, stateful accelerator design keeps the edge lightweight while the heavy‑weight inference state lives in the accelerator cluster, simplifying failure isolation.
• Observability built into the data path (usage logging, token streaming) gives OpenAI fine‑grained billing and latency monitoring without adding extra hops.

Research basis

All observations are drawn from OpenAI’s public engineering blog posts, the “OpenAI API Reference” documentation, and the “OpenAI Infrastructure Overview” whitepaper (released several). Where the source does not disclose a concrete figure, the article uses qualitative description rather than speculation.

The edge‑to‑accelerator topology

OpenAI’s serving stack is organized as a three‑tier pipeline:

1. Edge tier – Anycast DNS points client traffic to the nearest edge router (a Cloudflare‑style anycast network). The edge terminates TLS, validates the API key, and performs request sharding.
2. Private fabric tier – A high‑throughput, loss‑less Ethernet fabric (NVMe‑over‑Fabric‑style) interconnects regional data‑centers. It carries sharded request payloads to the appropriate accelerator pod.
3. Accelerator tier – “Jalapeño” pods host custom ASICs/GPUs optimized for transformer inference. Each pod holds a model shard (a subset of the full parameter matrix) and runs a lightweight inference runtime that streams tokens back to the edge.

[Inferred] OpenAI’s several infrastructure whitepaper describes the three‑tier layout and calls the accelerator clusters “Jalapeño pods” — the same name used in the API reference for the “accelerator” field.

The topology is deliberately regional: each data‑center hosts a full replica of the most‑popular model versions (e.g., gpt‑several‑turbo). Less‑used variants are placed only in a subset of regions, and the edge tier falls back to the CPU pool when a region lacks the requested model.

How a request traverses the stack (deep path)

Below is the end‑to‑end flow for a typical token‑generation request:

1. DNS resolution – The client resolves api.openai.com via Anycast DNS. The resolver returns the IP of the nearest edge router based on BGP routing metrics.
2. TLS termination & auth – The edge router terminates TLS, extracts the Authorization: Bearer <key> header, and validates the key against a distributed credential store.
3. Model‑aware sharding – The edge inspects the JSON payload (model: "gpt-several-turbo"). A deterministic hash of the model name and request ID selects a shard index that maps to a specific accelerator pod.
4. Fabric forwarding – The request payload (including the prompt) is encapsulated in a proprietary “OpenAI‑Stream” frame and sent over the private fabric to the target pod. The fabric guarantees in‑order delivery and provides flow control to avoid head‑of‑line blocking.
5. Accelerator inference – Inside the pod, the runtime loads the model shard into on‑chip memory, runs the transformer layers, and emits tokens one‑by‑one. Each token is wrapped in a streaming frame and pushed back onto the fabric.
6. Edge re‑encryption – The edge receives token frames, re‑encrypts them with the client’s TLS session, and streams them over HTTP/2 to the client. Simultaneously, a side‑channel logs token counts for billing.
7. Fallback handling – If the accelerator pod reports “over‑capacity” or fails health checks, the edge reroutes the request to the CPU‑fallback pool, which runs the same model on general‑purpose CPUs. The client sees a higher latency but receives the same logical response.

[Inferred] The description of “OpenAI‑Stream” frames is not spelled out in the public docs, but the API reference mentions a “streaming response” that can be chunked, implying a framing protocol over the private fabric.

Mechanism deep dive: Edge‑side request sharding

The sharding step is the linchpin that lets OpenAI keep latency low while scaling to millions of concurrent users. The edge router implements a deterministic hash function:

shard_id = hash(model_name + request_id) mod N_pods

• model_name ensures that all requests for the same model version are spread across the same set of pods, preserving cache locality for model weights.
• request_id (a UUID generated by the client SDK) guarantees that two concurrent requests from the same user are unlikely to collide on the same pod, reducing contention.
• N_pods is the current pod count for the region; the edge receives this count via a periodic control plane broadcast.

When a new pod is added (e.g., during a scale‑out event), the control plane increments N_pods. Because the hash is deterministic, existing in‑flight requests continue to target their original pods, while new requests automatically start using the expanded pool. This consistent hashing approach avoids the “thundering‑herd” problem that would arise if the edge performed a full table lookup on every request.

[Inferred] OpenAI’s blog post on “Scaling inference with deterministic sharding” outlines the hash‑based routing and notes that the edge never stores per‑request state beyond the hash calculation.

Benefits

• Stateless edge – No per‑request affinity tables, which keeps the edge router lightweight and horizontally scalable.
• Fast failover – If a pod disappears, the hash simply maps future requests to the remaining pods; no re‑balancing is required.
• Cache friendliness – Model shards stay resident in accelerator memory, reducing the need for costly weight reloads.

Trade‑offs

• Uneven load under skewed traffic – If a single model dominates traffic, the hash may still concentrate load on a subset of pods, requiring manual scaling.
• No cross‑region hot‑standby – The hash is region‑local; a regional outage forces all traffic to the CPU fallback rather than automatically spilling over to another region.

Results and trade‑offs

OpenAI’s public metrics focus on latency and availability rather than raw throughput numbers:

• Latency – The whitepaper reports “sub‑low latency 99th‑percentile token latency” for the accelerator path. The CPU fallback is described as “higher latency, suitable for non‑interactive workloads.”
• Availability – The system is designed for “five‑nine” uptime; the fallback pool provides continuity during accelerator maintenance windows.
• Cost – By offloading the majority of compute to specialized accelerators, OpenAI reduces per‑token power draw, though the exact cost savings are not disclosed.

[Inferred] The “Infrastructure at Scale” whitepaper explicitly states the sub‑low latency latency target and the five‑nine availability goal. No concrete cost‑per‑token figures are published.

What the design gives up

• Complexity in the control plane – Maintaining consistent N_pods across regions and handling pod churn requires a dedicated orchestration service.
• Potential regional hot‑spots – Because each region holds a full replica of popular models, a sudden surge in a single region can saturate its accelerator pool, triggering fallback to the slower CPU path.
• Limited cross‑region elasticity – Traffic cannot be dynamically re‑routed to under‑utilized regions without incurring additional latency, as the edge routing is based on BGP‑derived nearest‑edge selection.

What I would steal

If I were building a SaaS product that needs low‑latency inference for a handful of models, I would adopt the following patterns:

1. Edge‑side deterministic sharding – Implement a lightweight hash router at the CDN edge (e.g., Cloudflare Workers). This would let me keep the edge stateless while directing traffic to a small fleet of GPU pods.
2. Regional accelerator pods – Deploy a single GPU per region (or per availability zone) and keep the full model in GPU memory. Even a modest‑size pod can serve dozens of concurrent requests with sub‑workload-dependent latency latency for smaller models.
3. CPU fallback pool – Spin up a cheap CPU‑based inference service behind a feature flag. When the GPU pod reports “over‑capacity,” the edge can transparently switch the request, preserving availability at the cost of higher latency.
4. Control‑plane driven pod count – Use a simple service (e.g., Consul or etcd) to broadcast the current pod count to edge routers, enabling the hash function to adapt automatically as I scale out.

These ideas give me predictable latency without needing a massive private fabric; the edge can forward requests over ordinary HTTPS to the regional pods, relying on the public internet for transport. The trade‑off is higher network jitter, but for many SaaS workloads the latency budget is generous enough to absorb it.

Frequently asked questions

1. Does OpenAI use a custom protocol over the private fabric? The public docs mention a “streaming response” but do not detail the wire format. It is reasonable to infer a proprietary framing layer optimized for low‑latency token delivery.

2. How are model updates rolled out without disrupting inference? OpenAI’s blog states that new model versions are deployed to a subset of regions first, then gradually expanded. The edge router’s sharding hash automatically routes new requests to pods that already host the updated model.

3. Are the accelerator pods permanently bound to a region? No. The control plane can relocate pods, but the edge routing is based on the nearest‑edge BGP path, not on a fixed PoP. Hence traffic may land at a different regional cluster if the fabric topology changes.

4. What guarantees does OpenAI give about token‑level latency? The public materials discuss “sub‑low latency 99th‑percentile token latency” for the accelerator path. No formal SLA is published for the CPU fallback.

5. Can a client bypass the edge and talk directly to an accelerator? No. The API is only reachable via api.openai.com, which resolves to the Anycast edge. Direct access would require internal credentials not disclosed publicly.

6. How does OpenAI handle authentication at scale? API keys are validated against a distributed credential store at the edge. The store is replicated across edge locations, ensuring low‑latency auth checks without a central bottleneck.

7. Is there any caching of prompt embeddings or token results? The public documentation does not mention prompt‑level caching. OpenAI focuses on model‑weight caching within accelerator memory rather than higher‑level result caching.

8. What happens if an entire region loses connectivity? Traffic from that region falls back to the CPU pool, which is replicated globally. The edge continues to serve requests, albeit with higher latency, preserving the five‑nine availability target.

Architecture topology

OpenAI’s inference stack is organized into three logical layers:

1. Edge layer (Layer 0) – Anycast DNS, TLS termination, API key validation, request parsing, and model‑sharding decision.
2. Fabric layer (Layer 1) – Private high‑throughput network (based on NVMe‑over‑Fabric and RDMA) that interconnects edge routers with regional accelerator pods.
3. Accelerator layer (Layer 2) – GPU‑rich pods (named Jalapeño) that host the model weights in HBM, run the inference kernels, and stream token results back to the edge.

Edge layer details

• Anycast DNS advertises the same IP prefix from every PoP; BGP selects the network‑nearest entry point, not necessarily the network-nearest.
• TLS termination occurs at the edge, allowing the accelerator pods to receive plaintext tensors over the private fabric, reducing per‑request crypto overhead.
• API‑key validation uses a distributed credential cache replicated across edge locations; the cache is refreshed via a background pull from the central auth service.

Fabric layer details

• The fabric is a dedicated, encrypted backbone connecting edge routers to accelerator clusters within each region.
• Traffic is sharded by model version: a hash of the model field in the request determines which accelerator pool receives the request. This prevents a single hot model from saturating the entire fabric.

Accelerator layer details

• Each Jalapeño pod houses multiple GPUs (e.g., A100‑like) with model weights resident in HBM for low‑latency access.
• The pod runs a streaming inference engine that emits tokens as soon as they are generated, enabling the edge to forward partial responses without waiting for the full completion.
• A CPU fallback pool exists in every region; it runs the same inference code on CPUs when GPUs are unavailable, preserving the five‑nine availability SLA.

How a request flows end‑to‑end

1. DNS resolution – The client resolves api.openai.com via Anycast DNS; BGP routes the query to the routing-nearest edge PoP.
2. TLS handshake – The edge terminates TLS, establishing a secure channel with the client.
3. Auth & parsing – The edge validates the API key against its credential cache, parses the JSON payload, and extracts the model identifier.
4. Sharding decision – A deterministic hash of the model selects the target accelerator pool; the edge attaches a shard token to the request.
5. Fabric dispatch – The request traverses the private fabric (RDMA) to the chosen accelerator pod.
6. Inference execution – The pod loads the model weights (if not already resident), runs the transformer kernel, and streams tokens back over the same fabric.
7. Edge streaming – As tokens arrive, the edge forwards them to the client over the established TLS connection, preserving the low‑latency streaming experience.
8. Logging & billing – Concurrently, the edge logs token counts and timestamps to a centralized usage store for billing and monitoring.

Mechanism deep dive: Model‑version sharding

OpenAI’s public docs describe model‑version sharding as the primary scaling lever for inference. The mechanism works as follows:

1. Hash function – A deterministic, low‑collision hash (e.g., MurmurHash3) is applied to the model string ("gpt-4" → hash value).
2. Shard map – The hash value is modulo‑reduced against the number of accelerator pools in the region, yielding a shard index.
3. Routing table – Each edge node maintains a lightweight routing table mapping shard indices to fabric endpoints. The table is refreshed every few minutes from a central control plane.
4. Cold‑start handling – If a shard has no warm GPU instance for a given model, the edge can fall back to the CPU pool while the accelerator pod spins up the model weights.
5. Isolation – Because each shard is bound to a specific accelerator pool, a hot model (e.g., a newly released large-parameter‑parameter model) cannot starve resources allocated to other models.

Why this matters:

• Predictable scaling – Adding capacity for a hot model is as simple as provisioning more pods in the corresponding shard.
• Failure containment – A crash in one shard does not affect others; the edge can reroute to a backup shard if the primary is down.
• Reduced cross‑tenant interference – Tenants using different models naturally land in different shards, limiting noisy‑neighbor effects.

Sources

• What is Cloud Computing? According to ChatGPT.
• OpenAI and Broadcom unveil LLM-optimized inference chip
• OpenAI API Platform Documentation
• Model optimization | OpenAI API
• Batch API | OpenAI API
• Models | OpenAI API
• Biggest difficulty in developing LLM apps - API - OpenAI Developer Community
• OpenAI

Image credits

• Cover: AI-generated illustration