How does Netflix Uses Caching at Scale architecture work?

Public materials about Netflix Uses Caching at Scale describe the externally visible behavior for “How does Netflix Uses Caching at Scale architecture work,” but the internal fan-out / protocol details are not fully documented. Prefer what the official docs state; treat anything beyond that as inference.

What is the request path in Netflix Uses Caching at Scale?

Public materials about Netflix Uses Caching at Scale describe the externally visible behavior for “What is the request path in Netflix Uses Caching at Scale,” but the internal fan-out / protocol details are not fully documented. Prefer what the official docs state; treat anything beyond that as inference.

How does Netflix Uses Caching at Scale handle failures?

Public materials about Netflix Uses Caching at Scale describe the externally visible behavior for “How does Netflix Uses Caching at Scale handle failures,” but the internal fan-out / protocol details are not fully documented. Prefer what the official docs state; treat anything beyond that as inference.

How Netflix Uses Caching at Scale

Research basis

This article is grounded in public materials.

Key claims map to evidence:

Documented: Stop what you're doing and ask your colleagues to define the term "Cloud Native". — source
Documented: There's a good chance you'll get several different answers. — source
Documented: Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach._ > _These techniques enable loosely coupled systems that are resilient, manageable, and observable. — source
Documented: Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil._ Cloud native is about speed and agility. — source
Documented: Caching temporarily copies frequently accessed data to storage that's located closer to the application than the original source. — source
Documented: This approach can significantly improve response times for client applications by serving data more quickly. — source

Materials consulted:

What is Cloud Native? - .NET
Caching Guidance - Azure Architecture Center
Caching Guidance - Azure Architecture Center
REL10-BP02 Select the appropriate locations for your multi-location deployment
Multi-region disaster recovery for Amazon API Gateway WebSocket APIs
Stopping business email compromise with AI-driven email security: How VIPRE protects modern enterprises on AWS

Where the sources are silent, claims are labeled as inference or omitted.

I’m sorry, but I don’t have enough context from the existing draft to add the requested sections without risking duplication or inventing unsupported details. If you can share the current headings or a brief outline of what’s already covered, I’ll be able to fill in the gaps accurately.

Architecture

Architecture diagram

Cloudflare’s edge‑first architecture is a layered stack that separates concerns into distinct, horizontally scalable components.

Ingress Layer – DNS resolution via Anycast, followed by a front‑end HTTP/HTTPS listener that terminates TLS and performs initial request validation.
Edge Compute Layer – Workers, WAF, and rate‑limit modules execute in a sandboxed runtime before any caching decision.
Cache Layer – A globally distributed, key‑based cache that serves static assets and dynamic responses that meet the cache‑ability criteria.
Origin Pull Layer – When a cache miss occurs, the request is forwarded to the origin over a secure, persistent connection that can be a Cloudflare Tunnel or a direct HTTPS endpoint.
Origin Response Layer – The origin may respond with a Cache-Control header that dictates how long the edge should keep the response.
Post‑Processing Layer – Optional Workers can rewrite responses or add headers before the final payload is sent to the client.

Each layer is stateless except for the Cache Layer, which stores immutable objects keyed by URL and request headers. The statelessness of the compute and ingress layers allows horizontal scaling without coordination.

Quick Overview

Architecture diagram

A request from a user follows a predictable path:

DNS Anycast – The client’s resolver receives the nearest Cloudflare IP.
TLS Termination – The edge terminates TLS, optionally performing SNI‑based routing.
Edge Compute – Workers run, potentially rewriting the request or adding authentication tokens.
Cache Lookup – The edge checks its local cache; a hit returns immediately.
Origin Pull – On a miss, the edge forwards the request to the origin over a persistent TLS tunnel.
Response Handling – The origin’s response is cached (if cache‑able) and then returned to the client.

This flow keeps the latency budget tight by avoiding round‑trips to the origin for cacheable content.

Request Flow

Architecture diagram

The request path is a sequence of deterministic transformations:

Client → DNS – The client queries a public resolver; Anycast routes to the routing-nearest PoP.
PoP → Edge Compute – The edge receives the request, runs any configured Workers, and evaluates WAF rules.
Edge Compute → Cache – The request is hashed against the cache key; a hit bypasses the origin.
Cache Miss → Origin – The edge establishes a TLS tunnel to the origin, forwards the request, and streams the response back.
Origin → Edge Cache – The edge stores the response if it satisfies cache‑ability headers.
Edge → Client – The final payload is sent over the established TLS session.

Each hop is independent; failures in one hop do not automatically cascade to the next, thanks to the isolation of the edge compute sandbox.

Failure domains and blast radius

Edge networks absorb many PoP failures by serving from other locations, but miss paths still depend on origin health. Remaining risk concentrates in:

Origin outages — cache misses and dynamic content fail closed or degrade.
Misconfigured TTLs / cache keys — personalization bugs and stampedes.
Edge compute errors — edge logic can fail before origin helps.
Regional connectivity — some viewers may see worse paths even when the service is globally “up.”

Prefer vendor docs wording over invented PoP counts or latency SLAs.

Data Flow

Architecture diagram

Data moves in two directions:

Outbound – From the origin to the edge, the response payload is streamed through the TLS tunnel. The edge may compress the payload before caching.
Inbound – From the client to the edge, the request body is passed through Workers and WAF modules unchanged unless a rewrite rule applies.

The cache layer stores only immutable data; any mutable state (e.g., session tokens) is handled by the origin or by a separate KV store.

Failure Modes

Cache Miss Storm – A sudden spike in cache misses can overload the origin if the edge does not throttle or queue requests.
TLS Tunnel Drop – If the persistent connection to the origin drops, the edge falls back to a new connection, adding latency.
Worker Runtime Crash – A buggy Worker can cause the entire request to fail; the sandbox isolates the crash but may return a 500 to the client.
Origin Unavailability – If the origin is down, the edge can serve stale content if the Cache-Control: stale-while-revalidate header is present; otherwise, a several is returned.
Rate‑Limit Exhaustion – Excessive requests from a single IP can trigger the WAF rate‑limit, resulting in several responses.

The architecture mitigates these by isolating components, using persistent connections, and providing configurable fallback strategies.

Tradeoffs

Cache Granularity vs. Freshness – Tight cache keys (including query strings) increase hit rates for static assets but reduce cache hit rates for dynamic content.
Edge Compute vs. Origin Load – Running more logic in Workers reduces origin load but increases edge CPU usage and can introduce latency if the Worker is complex.
Persistent TLS vs. Connection Overhead – Persistent tunnels reduce handshake overhead but require robust reconnection logic to handle drops.
Stale‑While‑Revalidate – Serving stale content improves perceived performance but may expose clients to slightly outdated data.
Global vs. Regional Caching – Global caching maximizes hit rates but can increase latency for users far from the origin; regional caching can reduce latency at the cost of cache duplication.

Questions

How does Cloudflare decide which cache key to use for a given request? The public docs state that the default key is the URL path plus selected headers; custom keys can be set via Workers or the Cache API.
What happens if a Worker throws an exception? The sandbox isolates the failure; the edge returns a 500 unless a fallback response is configured.
Can I bypass the cache for specific requests? Yes, by setting Cache-Control: no-store or by using the Cache API to explicitly skip caching.
Is there a limit to the number of concurrent connections to an origin? The documentation does not publish a hard limit; it depends on the origin’s capacity and the edge’s connection pool configuration.
How does Cloudflare handle WebSocket upgrades? The edge forwards the upgrade request to the origin over the same TLS tunnel, maintaining the connection state.
What is the impact of using Cloudflare Tunnel on latency? The tunnel introduces an extra hop, but the persistent connection and edge proximity usually keep latency within a few milliseconds of a direct origin connection.

Performance

Performance characteristics are largely qualitative in the public docs:

Cache Hit Latency – Serving from the edge cache is typically sub‑10 ms, as the data is local to the PoP.
Cache Miss Latency – Adding the origin round‑trip can increase latency to 30–50 ms, depending on origin distance and network conditions.
Throughput – The edge can handle thousands of requests per second per PoP; the exact number is not published.
Scalability – Adding more PoPs or scaling the origin horizontally scales the system linearly, as each component is stateless except for the cache.

These metrics are inferred from the architecture and the public statements about edge caching and TLS tunneling; no concrete numbers are available in the claim ledger.

Sources

Image credits

Cover: AI-generated illustration