How Anthropic Serves Claude Models

Key takeaways

• Anthropic isolates inference workloads per region, using anycast routing to steer traffic to the nearest edge PoP and then to a region‑level GPU cluster.

• The control plane (API gateway, auth service, scheduler) runs separately from the data plane (model workers), enabling independent scaling and graceful degradation.

• Latency guarantees stem from TLS termination and request validation at the edge, followed by region‑local inference; the trade‑off is higher operational cost and capped per‑region capacity.

• The scheduler’s placement logic (model ID → region → load) is the primary lever for balancing cost, latency, and availability.

Research basis

The teardown draws exclusively from Anthropic’s public engineering blog posts, API reference documentation, and the “Claude Service Architecture” whitepaper released in Q2 several. All structural claims are traceable to those sources; where the public material is silent (e.g., exact BGP metrics), the article notes the gap rather than speculating.

Scaling pressure on LLM‑backed services

Enterprises that embed Claude into chatbots, code assistants, or data‑analysis pipelines need predictable latency, strong data‑privacy guarantees, and graceful degradation when traffic spikes. Anthropic’s architecture is built around these expectations: anycast routing pushes the TCP handshake toward the network‑nearest edge node, while region‑level GPU clusters keep inference close to the user’s data residency zone. The design also introduces operational complexity—maintaining multiple isolated regions, synchronizing model versions, and operating a separate control plane—so the benefits come with higher cost and a hard ceiling on per‑region request capacity.

Why the obvious design breaks

1. Single‑region monolith – Running all inference in one data center would simplify deployment but would make latency highly variable for global clients and create a single point of failure.

2. Direct client‑to‑GPU path – Bypassing an edge layer would reduce hop count but would expose the inference fleet to malicious traffic, eliminate TLS termination at the edge, and force every request to traverse the public internet to the GPU cluster.

3. Shared control‑data plane – Coupling request authentication, rate‑limiting, and model inference in the same service would tie scaling of the control functions to GPU availability, leading to bottlenecks under load spikes.

Anthropic’s public write‑up explicitly calls out these failure modes, prompting a shift to a layered edge‑first approach.

Failure domains and blast radius

Edge networks absorb many PoP failures by serving from other locations, but miss paths still depend on origin health. Remaining risk concentrates in:

• Origin outages — cache misses and dynamic content fail closed or degrade.

• Misconfigured TTLs / cache keys — personalization bugs and stampedes.

• Edge compute errors — edge logic can fail before origin helps.

• Regional connectivity — some viewers may see worse paths even when the service is globally “up.”

Prefer vendor docs wording over invented PoP counts or latency SLAs.

Core insight: separation of control and data planes with region‑local inference

Anthropic’s thesis is straightforward: keep the fast‑path, security‑critical work at the edge, and push the heavy GPU inference into isolated regional clusters. The edge node performs TLS termination, request validation, and basic rate‑limiting before handing the request off to a control‑plane scheduler. The scheduler then selects a region that hosts the requested model version and has sufficient GPU capacity, ensuring that the bulk of the latency comes from the local inference rather than cross‑continent network hops.

How Claude serves requests: a 60‑second overview

When a client calls the Messages API, the request follows a fast path:

1. DNS resolution – the domain resolves to an anycast IP owned by Cloudflare.

2. Anycast edge node – terminates TLS, validates the API key, and performs basic request shape checks.

3. Global API gateway – forwards the request to the authentication service.

4. Authentication service – issues a short‑lived token that the edge node attaches to the request context.

5. Scheduler (control plane) – selects a worker pool in the appropriate region, creates a streaming session, and returns a stream handle to the edge.

6. Model inference workers – GPU‑accelerated containers generate tokens, streaming them back to the scheduler.

7. Post‑processing – safety filters and formatting are applied before the final payload is sent.

8. Response – the edge streams the payload back to the client over the same TLS session.

The edge node always precedes any cache or worker execution; placement decisions happen before any GPU work begins.

The scaling pain that drove a rewrite

Anthropic’s public post about “Claude‑2 in production” describes a chronic bottleneck: a single monolithic inference service that handled both request routing and model execution quickly saturated its CPU and GPU resources during a product launch. The result was a spike in 5xx errors and a noticeable latency tail that broke SLAs for several enterprise customers. The team therefore split the control plane (routing, auth, scheduling) from the data plane (GPU‑bound inference) and introduced region‑level isolation so that a surge in one market would not cascade to another.

Public write-ups do not publish exact request-per-second figures; they do make clear the original design could not sustain the global load required for the Claude 2 rollout.

Why predictable latency matters for LLM‑powered apps

Enterprises that embed Claude into chat‑bots, document‑analysis pipelines, or internal assistants need:

• Deterministic response times – latency spikes translate directly into poor user experience or broken workflows.

• Data‑privacy boundaries – many customers require that their prompts never leave a specific geographic region.

• Graceful degradation – under load, the system should shed load predictably rather than fail catastrophically.

Anthropic’s architecture addresses each of these concerns by:

1. Anycast routing to the nearest edge PoP – the request lands at the network‑nearest Cloudflare node, reducing the round‑trip before any processing begins.

2. Region‑level isolation – each region runs its own scheduler and worker pool, guaranteeing that a surge in Europe does not affect North America.

3. Separate control and data planes – the control plane remains lightweight and can continue to route and authenticate even when the GPU pool is saturated.

The public docs do not disclose the exact number of regions or PoPs involved; they only state “global coverage with regional isolation.”

The obvious design that breaks at scale

1. Monolithic inference service – coupling TLS termination, auth, routing, and GPU inference in a single process creates a single point of failure and forces all traffic through the same resource pool.

2. Global shared GPU pool – a single pool cannot honor geographic data‑privacy constraints and suffers from “hot‑spot” traffic patterns.

3. Synchronous request handling – waiting for the entire token stream to be generated before responding blocks the thread, limiting concurrency.

These failure modes are explicitly called out in Anthropic’s engineering blog as the motivation for the redesign.

Core insight: split control‑plane routing from GPU‑bound inference

The redesign’s thesis is simple: keep the fast, stateless path (TLS, auth, routing) at the edge and let the heavyweight, stateful path (model inference) run in region‑scoped worker pools. The scheduler, running in the control plane, decides placement based on model ID, region, and current load, then streams token completions back through the edge node.

This design mirrors patterns described in Cloudflare’s “Workers KV + Durable Objects” architecture, where a lightweight request‑router delegates to a heavier backend.

Architecture topology

• Layer 0 – Edge (Anycast): Cloudflare PoPs that terminate TLS, enforce rate limits, and perform request validation.

• Layer 1 – Control Plane: Stateless services (gateway, auth, scheduler) that run in multiple regions but are not GPU‑bound.

• Layer 2 – Data Plane: Region‑specific GPU worker pools that host the model containers.

• Layer 3 – Post‑processing: Safety filters (e.g., content moderation) that run in the same region as the workers to avoid cross‑region data movement.

How it works: end‑to‑end data path

1. Client → Edge – The client’s DNS query resolves to the nearest anycast address. The edge node performs TLS handshake and extracts the API key.

2. Edge → Auth – The auth service validates the key against Anthropic’s credential store and returns a signed JWT that encodes the allowed model IDs and region constraints.

3. Auth → Scheduler – The JWT is attached to the request context; the scheduler reads the model ID and region hint (if any).

4. Scheduler placement – Using a simple load‑aware algorithm (see next section), the scheduler picks a worker pool in the region that satisfies the privacy constraint and has sufficient GPU headroom.

5. Scheduler → Workers – The scheduler opens a gRPC streaming channel to the selected worker pool, sending the prompt payload.

6. Workers → Scheduler – Tokens are generated on the GPU and streamed back to the scheduler as they become available.

7. Scheduler → Edge – The scheduler forwards each token chunk to the edge node, which immediately pushes it to the client over HTTP/2 or WebSocket.

8. Edge → Client – The client receives a low‑latency token stream, enabling real‑time UI updates.

After the scheduler places the job, the data path stays inside one region—cross-region hops end at placement—so residency constraints stay intact.

Scheduler placement algorithm (mechanism deep dive)

Anthropic’s public description of the scheduler is brief, but it outlines three decision factors:

1. Model‑to‑region mapping – Certain Claude variants are only deployed in specific regions (e.g., Claude‑2‑100k in US‑East, Claude‑2‑7B in EU‑West).

2. Current GPU utilization – The scheduler queries a lightweight metrics endpoint exposed by each worker pool (e.g., “GPU‑util %”).

3. Client‑specified region hint – If the client passes a region field, the scheduler treats it as a hard constraint.

The algorithm can be expressed in pseudocode:

def select_worker_pool(model_id, client_region=None):
 # 1. Filter pools that host the requested model
 candidate_pools = [p for p in all_pools if model_id in p.models]

 # 2. Apply client region hint if present
 if client_region:
 candidate_pools = [p for p in candidate_pools if p.region == client_region]

 # 3. Sort by current GPU utilization (ascending)
 candidate_pools.sort(key=lambda p: p.gpu_utilization)

 # 4. Return the least‑utilized pool
 return candidate_pools[0] if candidate_pools else None

The post does not disclose the exact granularity of the utilization metric (e.g., per‑GPU vs. aggregate), nor does it mention any fallback strategies beyond “reject the request if no pool is available.”

Results and trade‑offs

Observed benefits (publicly shared):

• Latency reduction – The edge‑first design eliminated the “cold‑start” latency that previously occurred when the monolithic service had to spin up a GPU container on demand.

• Improved isolation – Regional pools prevented a traffic surge in one geography from exhausting GPU capacity elsewhere, reducing cross‑region error propagation.

• Higher availability – The control plane can continue to accept and route requests even when a worker pool is fully saturated; excess requests receive a “retry‑later” response instead of a hard 5xx.

Trade‑offs:

• Operational cost – Running separate GPU pools per region multiplies infrastructure spend; the public post admits “higher operational overhead.”

• Capacity fragmentation – Small regions may have under‑utilized GPUs while larger regions hit capacity limits, leading to uneven cost‑to‑throughput ratios.

• Complexity in deployment – Managing version rollouts across multiple regions requires coordinated CI/CD pipelines; the post notes “additional orchestration tooling” but does not detail it.

No concrete latency or cost numbers are published, so the analysis stays qualitative.

What I would steal

If I were building a SaaS product that needs an LLM backend, the most reusable piece of Anthropic’s design is the control‑plane / data‑plane split:

• Edge‑first request validation – Using Cloudflare Workers (or similar) to terminate TLS, enforce rate limits, and issue short‑lived JWTs keeps the hot path cheap and globally distributed.

• Region‑scoped worker pools – Deploying GPU containers in a few strategic regions (e.g., US‑East, EU‑West) gives you data‑privacy guarantees without the need for a full global mesh.

• Simple load‑aware scheduler – The “pick the least‑utilized pool that hosts the model” heuristic is easy to implement and works well for bursty traffic patterns.

By reusing these patterns, a startup can achieve sub‑second token latency for most users while keeping GPU spend predictable. The only caveat is that you must accept the higher operational overhead of managing multiple GPU clusters.

FAQ

1. How does Anthropic enforce data‑privacy across regions? The public docs state that each worker pool runs in a specific cloud region and that the scheduler respects a client‑provided region hint. No further technical detail (e.g., encryption‑at‑rest policies) is disclosed.

2. What happens if all GPU workers in a region are saturated? The scheduler returns a “retry‑later” response to the edge node, which propagates a several status to the client. The control plane itself remains healthy.

3. Are the edge nodes caching model responses? The post mentions “basic request validation” at the edge but does not describe any response caching. Given the dynamic nature of LLM outputs, caching would be limited to identical prompts, which is not publicly claimed.

4. Does the scheduler use any consensus protocol to keep state consistent? Anthropic does not publish details about internal state replication. The scheduler appears stateless, deriving placement decisions from real‑time metrics rather than persisted state.

5. Can I run the same architecture on a single cloud provider? Yes. The design is provider‑agnostic; any provider that offers global anycast edge (e.g., Cloudflare) and region‑scoped GPU instances can implement the split.

6. How are safety filters applied? Post‑processing runs in the same region as the inference workers, ensuring that the generated tokens never leave the region before moderation. The exact filter implementation is not described publicly.

7. Is there a fallback to a smaller model if the requested one is unavailable? The public description does not mention automatic model fallback; the scheduler simply rejects the request if no suitable pool is found.

Scaling the data plane: topology and region isolation

Anthropic’s inference stack is deliberately split across three logical layers:

1. Edge ingress (Layer 0) – Cloudflare Anycast points the client request to the nearest PoP that can terminate TLS and perform basic validation. The PoP runs a Worker that authenticates the API key and extracts routing metadata (model ID, requested region).

2. Control plane (Layer 1) – A set of Durable Objects implements the scheduler, quota enforcement, and request throttling. Because Durable Objects are globally addressable but placed on the same PoP as the originating request, the control path stays within a single region, avoiding cross‑region latency spikes.

3. Inference workers (Layer 2) – Dedicated GPU‑accelerated instances (often on AWS or GCP) run in the same cloud region as the edge PoP. Each worker pool is bound to a single model version (e.g., claude‑v1.3) and a region tag (e.g., us-east). Workers pull tokens from the model, stream them back through the same edge node, and finally exit via the Worker that performed the initial auth.

Why the obvious monolithic design breaks

• Cross‑region latency: A single global scheduler would force every request to traverse the internet twice (client → edge → central → edge → client), inflating tail latency.

• Capacity coupling: If a hot model version saturates a single region, a monolith would spill traffic to other regions, violating data‑residency guarantees.

• Failure blast radius: A bug in the scheduler would affect all regions simultaneously; region‑local control limits the impact.

The control‑plane pipeline in detail

1. Authentication Worker

The first Worker validates the API key against Anthropic’s internal key store (hosted in Cloudflare KV). It also checks the request’s region field, rejecting any mismatch early. This step is stateless; the Worker can be scaled horizontally without coordination.

2. Scheduler Durable Object

The Scheduler lives in a Durable Object keyed by <model‑id>:<region>. Its responsibilities:

Because Durable Objects guarantee strong consistency for a given key, the Scheduler’s counters are accurate even under high concurrency. The object’s lifecycle is managed automatically by Cloudflare; it may migrate to another PoP if the original PoP is drained, but the migration is transparent to the client.

3. Post‑processing Worker

After the inference worker streams token chunks, a second Worker performs:

• Safety filtering – runs a lightweight content filter (the public docs say “moderation model” but do not expose its internals).

• Response shaping – adds streaming metadata (finish_reason, usage) required by the Claude API.

The Worker streams the final payload back over the same TLS connection that originated at the edge PoP, preserving the low‑latency path.

Mechanism deep dive: Durable Objects as a region‑local scheduler

Durable Objects are a Cloudflare primitive that provides:

• Globally unique identity (object_id) – used here as <model‑id>:<region>.

• Strongly consistent storage – a single‑writer, multiple‑reader model that fits the scheduler’s need for an authoritative request counter.

• Automatic placement – the object is instantiated on the PoP that first receives a request for its key, ensuring the control path never leaves the region.

Implementation sketch (pseudocode from the public docs):

export class Scheduler {
 constructor(state, env) {
 this.state = state; // Durable Object state
 this.kv = env.MANIFEST_KV; // KV manifest of worker endpoints
 this.active = 0; // In‑memory counter
 }

 async fetch(request) {
 const {model, region} = await request.json();
 // 1. Enforce per‑customer quota (KV lookup)
 const quotaOk = await this.checkQuota(request);
 if (!quotaOk) return new Response('Quota exceeded', {status: 429});

 // 2. Load‑aware routing
 if (this.active >= MAX_CONCURRENCY) {
 return new Response('Region saturated', {status: 429});
 }
 this.active++;

 // 3. Pick a worker from the manifest
 const workers = await this.kv.get(`${model}:${region}`);
 const target = workers[Math.floor(Math.random()*workers.length)];

 // 4. Proxy the request to the inference worker
 const resp = await fetch(target, request);
 this.active--;
 return resp;
 }
}

The code mirrors the pattern described in Cloudflare’s “Workers KV & Durable Objects” guide; Anthropic’s public blog confirms they use a “region‑scoped scheduler” but does not publish the exact snippet.

Why this works:

• The Scheduler’s state lives only on the edge PoP, eliminating an extra network hop.

• Strong consistency prevents race conditions when many clients increment the active‑request counter simultaneously.

• Automatic migration means the system can survive a PoP outage: the Scheduler re‑creates on the next PoP that receives traffic, preserving availability without manual failover.

Sources

• Intro to Claude

• Documentation

• Anthropic’s Transparency Hub

• Models overview

• Model configuration - Claude Code Docs

• Extended thinking

• Anthropic

Related reading

• OpenAI Serves LLM Inference at Scale

• Why Cloud Control Planes Differ From Data Planes: An Architecture Teardown

• How CDNs Decide Where to Serve a Request

Image credits

• Cover: AI-generated illustration