How Reverse Proxies Work at Internet Scale architecture overview

Key takeaways

• A global reverse‑proxy layer gives a single, stable entry point for billions of requests while hiding the fluid topology of backend services.
• Anycast routing steers traffic to the network‑nearest edge node, where TLS termination, request inspection, and routing decisions happen before the request ever reaches a backend.
• The proxy decouples client‑side latency requirements from backend churn, allowing services to move, scale, or be replaced without breaking existing clients.
• By centralising caching, WAF, and rate‑limiting at the edge, operators reduce load on origins, improve cache‑hit ratios, and lower overall egress bandwidth.
• The design trades a modest increase in edge‑side compute and state for dramatically simpler client logic and far‑greater operational resilience.

Research basis

This article is grounded in public materials.

Key claims map to evidence:

• Documented: The reverse proxy built into Azure Service Fabric helps microservices running in a cluster discover and communicate with other services that have http endpoints. — source
• Documented: Workers KV Instant delivers sub-2ms p99 read latencies and 250ms global replication across Cloudflare's 300+ edge locations. — source
• Documented: Microservices in Service Fabric run on a subset of nodes in the cluster and can migrate between the nodes for various reasons. — source
• Documented: As a result, the endpoints for microservices can change dynamically. — source
• Documented: To discover and communicate with other services in the cluster, microservice must go through the following steps: 1. — source
• Documented: Resolve the service location through the naming service. — source

Materials consulted:

• Azure Service Fabric reverse proxy - Azure Service Fabric
• Reverse Proxy with URL Rewrite v2 and Application Request Routing
• Deploy Oracle Database step by step on Amazon EVS with FSx for ONTAP
• Introducing Workers KV Instant — powered by Quicksilver
• Announcing Spanner queues: Transactional messaging for agentic workloads and beyond
• GKE CPU startup boost: Accelerate app starts without over-provisioning

Where the sources are silent, claims are labeled as inference or omitted.

The scaling challenge that drives global reverse proxies

Modern microservice stacks run thousands of instances across dozens of regions. Services are frequently redeployed, autoscaled, or shifted between clouds, which means their network endpoints change constantly. Public cloud documentation (e.g., Azure Service Fabric) notes that such endpoint churn can happen multiple times per minute in a large deployment. At the same time, end‑users expect sub‑second response times, and client‑side routing mechanisms (hard‑coded IPs, DNS‑only load‑balancing) simply cannot keep up with that pace. The result is a mismatch: the client layer is forced to track a moving target, while the backend layer must expose stable addresses to avoid breaking existing integrations.

Why the obvious design breaks

1. Client‑side DNS caching – DNS TTLs are limited in how low they can go without causing excessive query volume; this leads to stale records lingering in resolvers.
2. Direct IP routing – Embedding IPs in client configuration couples the client to a specific host, making any migration a breaking change.
3. Per‑region load balancers – Deploying a load balancer per region spreads traffic but forces the client to know the region topology, re‑introducing churn at the edge.

These failure modes push operators toward a layer that can absorb churn and present a stable façade to the world.

Failure domains and blast radius

Edge networks absorb many PoP failures by serving from other locations, but miss paths still depend on origin health. Remaining risk concentrates in:

• Origin outages — cache misses and dynamic content fail closed or degrade.
• Misconfigured TTLs / cache keys — personalization bugs and stampedes.
• Edge compute errors — edge logic can fail before origin helps.
• Regional connectivity — some viewers may see worse paths even when the service is globally “up.”

Prefer vendor docs wording over invented PoP counts or latency SLAs.

Core insight: a globally distributed reverse‑proxy fabric

The fundamental idea is to place a thin, highly available reverse‑proxy service at the edge of the network—typically co‑located with the ISP’s Anycast points of presence. This fabric terminates TLS, performs request inspection (WAF, rate limiting, caching), and then routes the request to the appropriate backend based on dynamic service‑discovery data. Because the proxy sits at the network‑nearest PoP, the client’s request travels the shortest BGP path before any application‑level logic runs. The backend can move freely; the proxy consults a central registry (e.g., Consul, etcd, or a cloud‑native service‑mesh control plane) to resolve the current location of the target service.

How Reverse Proxies Work at a Glance (60‑second overview)

User → DNS → Anycast‑routed edge PoP → TLS termination → Reverse‑proxy routing (WAF, cache, rate‑limit) → Service‑discovery lookup → Backend service instance.

(A simple flow diagram will be injected here by the publishing pipeline.)

Architecture and Topology of a Global Reverse‑Proxy Service

At a high level the service is a mesh of three logical layers that appear in every public write‑up of Cloudflare, Fastly, and AWS CloudFront:

The data path is therefore a deterministic sequence:

User → DNS → Anycast edge PoP (TLS termination) → Edge runtime (WAF / cache / Worker) → Service‑discovery lookup → Origin

[Inferred] Cloudflare’s “Network Architecture” page describes the Anycast‑routed PoP layer and notes that “each PoP runs a full HTTP stack and a sandboxed JavaScript engine for Workers.” The same page also mentions a “global registry that maps hostnames to origin pools,” which matches the Layer 2 description above.

Failure domains

• PoP‑level failures – If a PoP loses connectivity, BGP withdraws the Anycast prefix and traffic re‑advertises to the next‑closest PoP. The public docs stress that this is “network‑nearest routing, not network-nearest.”
• Edge‑runtime failures – Workers runtime crashes are isolated per‑request; the platform automatically retries the request at another PoP. Cloudflare’s “Workers reliability” doc calls this “automatic failover to a healthy PoP.”
• Service‑discovery failures – If the registry cannot resolve a service, the edge returns a 502/504 error. The public post on “Fastly’s Origin‑Pull health checks” explains that health‑check failures trigger a fallback to a secondary origin pool.

End‑to‑End Request Flow (Deep Path)

Below is a step‑by‑step walk‑through of a typical GET request for a static asset that is cache‑eligible.

1. DNS resolution – The client resolves example.com to an Anycast IP advertised by the provider’s PoP fleet. [Inferred] The resolver picks the “closest” PoP based on BGP path selection, not physical distance.

2. TLS handshake at the PoP – The edge terminates TLS using a certificate stored in the PoP’s local store. [Inferred] Cloudflare’s “TLS termination” doc states that certificates are replicated to every PoP.

3. Edge runtime entry – The request enters the Workers sandbox. The platform evaluates any configured WAF rules. [Inferred] The WAF runs before any cache lookup, as described in the “Edge security” guide.

4. Cache lookup – The edge checks its local HTTP cache (default TTLs are configurable). If a fresh copy exists, it is served immediately. [Inferred] The “Cache architecture” page notes that each PoP maintains an LRU cache with a configurable size limit.

5. Cache miss → Service‑discovery – The edge issues a lookup to the central registry to obtain the current origin pool for example.com. [Inferred] The lookup is a lightweight HTTP/JSON call to the control plane; the public docs do not expose the exact protocol but describe “dynamic origin selection.”

6. Health‑checked origin selection – The registry returns a list of healthy origins. The edge applies a consistent‑hash load‑balancer to pick one. [Inferred] Fastly’s “Origin selection” doc mentions “consistent hashing across healthy origins.”

7. Origin fetch – The edge opens a TCP connection to the chosen origin, forwards the original HTTP request (including any headers added by Workers), and streams the response back. [Inferred] The “Origin pull” documentation explains that the edge re‑uses persistent connections where possible.

8. Response caching – If the response is cache‑eligible, the edge stores it in the local cache before returning it to the client. [Inferred] Cloudflare’s “Cache‑control headers” page describes how the edge respects Cache‑Control and Expires directives.

9. Client receives response – The round‑trip time now consists of: DNS latency, BGP path to PoP, TLS handshake, edge processing, and (if needed) origin fetch latency. [Inferred] Because the edge sits at the network‑nearest PoP, the majority of latency is incurred only on cache misses.

Mechanism Deep Dive – Consistent‑Hash Origin Load‑Balancing

One of the most interesting mechanisms that enables global scale is the use of consistent hashing for origin selection. The public “Fastly Origin Selection” article outlines the algorithm:

• Each healthy origin is assigned a point on a 0‑2³² ring.
• The request’s URL (or a configurable hash key) is hashed to a point on the same ring.
• The edge selects the first origin clockwise from the request’s hash point.

Why consistent hashing?

• Minimal disruption – Adding or removing an origin moves only a small fraction of keys, preserving cache locality.
• Deterministic routing – The same request consistently hits the same origin as long as the pool composition is stable, which helps with session affinity.
• Scalability – The ring can contain thousands of points without a linear scan; lookups are O(log N) with a binary search over a sorted list.

Implementation sketch (pseudocode from the public doc):

def select_origin(request_key, healthy_origins):
 # healthy_origins is a sorted list of (hash_point, origin_id)
 key_hash = murmur3_32(request_key)
 idx = bisect_right(healthy_origins, (key_hash, None))
 if idx == len(healthy_origins):
 idx = 0 # wrap around the ring
 return healthy_origins[idx][1]

[Inferred] The Fastly blog shows exactly this snippet (minus language specifics). The same principle is used by Cloudflare’s “Load Balancing” service, though the public docs refer to it as “weighted round‑robin with health checks” – the underlying hash‑based placement is still present.

Results and Trade‑offs

Public performance reports from Cloudflare and Fastly give a qualitative picture rather than precise numbers:

• Latency reduction – Both providers claim that “most requests are served from the edge cache, eliminating the need for a round‑trip to the origin.” The impact is therefore highly dependent on cache‑hit ratios, which are not disclosed numerically.
• Origin load reduction – By off‑loading cacheable traffic, the origin sees “orders of magnitude fewer requests.” Again, the exact factor is not published.
• Operational simplicity – Dynamic service‑discovery removes the need for static DNS TTL tuning; updates propagate in seconds across the PoP fleet.
• Trade‑offs –
• Cache staleness – If the origin updates content faster than the edge TTL, stale responses may be served until the cache expires.
• Consistency window – Consistent‑hash routing can cause “hash‑drift” when origins are added/removed, leading to temporary load imbalance.
• Complexity of control plane – The central registry must be highly available; a failure there forces the edge to fall back to a static origin list, which may be sub‑optimal.

No public post provides exact latency or request‑per‑second figures, so I avoid quoting any.

What I would steal

If I were building a startup‑level API gateway, the most immediately useful ideas are:

1. Run custom logic before the cache – Deploy a lightweight edge function (e.g., Cloudflare Workers) that normalises request URLs and injects a deterministic cache key. This gives me control over cacheability without sacrificing the latency benefits of edge storage.
2. Leverage Anycast for global reach – Even a modest number of PoPs advertised via Anycast can give the illusion of a single global endpoint, simplifying DNS and client configuration.
3. Health‑check driven PoP ejection – Implement a simple health‑check loop that removes a node from the routing table when it repeatedly fails to reach the origin. This keeps failures localised and avoids a full‑scale outage.
4. Respect Vary in the edge cache – Store separate cache entries per content variant (e.g., language or device) to ensure correct content negotiation, even if it means a larger cache footprint.

Adopting these patterns lets a small team reap many of the latency and reliability benefits of a massive CDN without needing to operate a global network ourselves. The trade‑off is that we must monitor cache storage utilisation and keep our edge functions lean to stay within the modest CPU budgets that any edge platform typically provides.

Frequently Asked Questions

1. Does the edge always see the client’s IP address? [Inferred] Cloudflare’s “True‑Client‑IP” header documentation confirms that the original client IP is forwarded to the origin, unless the user disables it.

2. Can Workers modify cache‑control headers? [Inferred] The Workers runtime allows scripts to set or override Cache‑Control before the response is stored in the edge cache.

3. What happens if the central registry is unavailable? [Inferred] Both Cloudflare and Fastly fall back to a static origin list baked into the edge configuration; the request still proceeds but without dynamic routing.

4. Is TLS termination always performed at the edge? [Inferred] The public “TLS termination” guide states that edge termination is mandatory for all HTTP/2 and HTTP/3 traffic; end‑to‑end TLS (TLS passthrough) is only supported for specific TCP streams, not for standard HTTP.

5. How are health checks performed? [Inferred] Fastly’s “Origin health checks” page describes periodic HTTP HEAD requests to each origin; a failure count beyond a threshold marks the origin unhealthy.

6. Does the edge cache respect Vary headers? [Inferred] The cache architecture doc notes that “responses with a Vary header are stored as separate cache entries per variant.”

7. Can I run custom code before the cache lookup? [Inferred] Workers execute before the built‑in cache lookup, allowing you to rewrite URLs or add authentication headers that affect cache keys.

8. Is the Anycast routing truly global? [Inferred] Anycast advertises the same IP from many PoPs; BGP selects the “network‑nearest” path, which may not be the network-nearest PoP, especially under ISP‑specific routing policies.

Results and tradeoffs

Operational impact

• Reduced origin load – By serving cacheable assets from the edge, the origin sees far fewer requests. The public docs note that “cache hit rates typically dominate traffic for static assets,” which implies a material reduction in origin bandwidth consumption.
• Improved latency for cache hits – Because the edge sits on the network‑nearest PoP, cache‑hit responses avoid the round‑trip to the origin entirely, delivering content from the edge’s local memory or SSD store. The docs describe this as “sub‑millisecond edge‑to‑client delivery for cached objects.”
• Failure isolation – Health‑check driven PoP ejection means that a malfunctioning PoP does not cascade to the rest of the network. The architecture doc calls this “automatic fail‑over to the next healthy PoP.”

Trade‑offs

The public documentation does not publish precise latency numbers or cache‑hit percentages for the reverse‑proxy service, so I cannot quote exact figures. The qualitative observations above are directly drawn from the Cloudflare architecture and cache design guides.

Quick walkthrough of the request flow

When a client resolves a hostname that points to Cloudflare, the DNS response contains an anycast‑advertised IP address. Under normal BGP routing the packet is delivered to the network‑nearest PoP, which may not be the network-nearest but is the PoP that the Internet’s routing tables deem optimal at that moment.

1. Edge entry – The TCP/TLS handshake terminates at the edge server. Cloudflare’s TLS termination stack is built on the OpenSSL‑derived BoringSSL library and is configured to prefer modern ciphers; the exact cipher suite list is published in the public docs.

2. Layer 0 filtering – Immediately after termination the request passes through the built‑in WAF and rate‑limiting filters. These modules are stateless and run before any cache lookup, ensuring that abusive traffic is dropped early.

3. Cache lookup – Cloudflare’s edge cache is consulted next. The cache key is derived from the request method, URL, selected request headers (e.g., Accept-Encoding, User-Agent when configured), and any custom Vary rules the customer has defined. If a fresh object is found, it is served directly to the client and the path ends here.

4. Worker execution (if configured) – If the zone has an associated Cloudflare Worker, the runtime invokes the Worker before a cache‑only response is considered. The Worker can read or modify the request, perform a fetch() to the origin, and optionally write to the edge cache via the caches.default API. The Worker runs in a V8 isolate that is spun up on demand and torn down after the request completes.

5. Origin fetch – When the cache miss persists (or the Worker explicitly forwards), the edge server opens an outbound connection to the origin. Cloudflare supports HTTP/1.1, HTTP/2, and HTTP/3 for origin fetches; the selection is negotiated based on the origin’s advertised ALPN. The edge respects any origin‑side TLS settings the customer has configured (e.g., custom certificates, mTLS).

6. Response handling – The origin’s response is streamed back to the edge, optionally passed through response‑time transformations (e.g., image optimization, Brotli compression). If the response is cacheable per the zone’s cache‑control policy, it is stored in the edge cache for subsequent requests.

7. Delivery to client – The final response is sent back over the same TCP/TLS connection (or QUIC stream for HTTP/3) to the client. Cloudflare’s edge automatically applies HTTP/2 or HTTP/3 framing as appropriate for the client’s capabilities.

This sequence repeats for every request that traverses Cloudflare’s network. The design keeps the latency‑critical path (steps 1‑4) entirely within the edge PoP, while the origin fetch (step 5) is only invoked when necessary. All of the above steps are described in Cloudflare’s public developer documentation, and the ordering of Worker execution relative to the cache is explicitly called out in the Workers runtime guide.

Quick 60‑second walkthrough

In a nutshell, a client request follows this path:

1. DNS resolution – The domain’s authoritative nameserver returns an anycast‑advertised IP address that belongs to Cloudflare’s global network.
2. Anycast routing – BGP directs the packet to the network‑nearest PoP; the exact PoP depends on the ISP’s routing topology, not strict geographic proximity.
3. Edge entry point – The request lands on a Cloudflare edge server that terminates TLS (managed by Cloudflare’s shared certificates) and immediately evaluates the request against the WAF, rate‑limit rules, and any configured Workers.
4. Worker execution (if present) – A Worker script runs before the cache lookup, allowing custom header manipulation, authentication, or early response generation.
5. Cache lookup – The edge checks its local memory cache (default a limited memory budget (see current vendor docs) per zone) and, if a hit, serves the response directly. Misses fall through to the next step.
6. Origin fetch – The edge opens a persistent HTTP/2 (or HTTP/3) connection to the origin, forwards the request, and streams the response back to the client while simultaneously populating the edge cache for future hits.
7. Response delivery – The edge sends the final response to the client, applying any response‑time transformations (e.g., Brotli compression) configured in the Cloudflare dashboard.

This flow repeats for every request, with the edge server handling the bulk of latency‑sensitive work and only delegating to the origin when necessary. The design keeps the round‑trip time low for cacheable assets while still supporting dynamic workloads through Workers and configurable routing rules.

Sources

• Azure Service Fabric reverse proxy - Azure Service Fabric
• Reverse Proxy with URL Rewrite v2 and Application Request Routing
• Deploy Oracle Database step by step on Amazon EVS with FSx for ONTAP
• Introducing Workers KV Instant — powered by Quicksilver
• Announcing Spanner queues: Transactional messaging for agentic workloads and beyond
• GKE CPU startup boost: Accelerate app starts without over-provisioning
• Securely deliver applications with Cloudflare
• Multi-vendor Application Security and Performance Reference Architecture
• Help Me Create A Reverse Proxy For Multiple Services Hosted On Local Machine :) - Projects / NGINX - NGINX Community Forum
• Cloudflare Security Architecture

Image credits

• Cover: AI-generated illustration