How API Gateways Route Requests Globally

Key takeaways

• Global routing is driven by latency, regulatory compliance, and high‑availability needs.
• A single‑region API endpoint can add 40–80 ms for users on the other side of the planet.
• Anycast DNS and edge‑first gateways let providers satisfy geo‑policy while keeping the control plane centralized.
• Classic L4/L7 load balancers fall short when traffic must respect data residency and failover across continents.
• The modern pattern is a tiered edge topology: Anycast DNS → edge POP → gateway node → policy‑aware routing logic → backend.

Research basis

This article is grounded in public materials.

Key claims map to evidence:

• Documented: An API gateway provides a centralized entry point for managing interactions between clients and application services in a microservices architecture. — source
• Documented: An API gateway helps address challenges such as determining what endpoints to call when new services are introduced or refactored, and handling concerns like SSL termination, mutual TLS, and authentication. — source
• Documented: The Gateway Routing pattern is used to route requests to multiple services or multiple service instances using a single endpoint. — source
• Documented: The Gateway Routing pattern is useful for exposing multiple services on a single endpoint and routing to the appropriate service based on the request. — source
• Documented: Title: API gateways - Azure Architecture Center URL Source: https://learn.microsoft.com/en-us/azure/architecture/microservices/design/gateway Markdown Content: In a microservices architecture, a client might interact with more than one front-end service. — source
• Documented: Given this fact, how does a client know what endpoints to call? — source

Materials consulted:

• API gateways - Azure Architecture Center
• Create routes for HTTP APIs in API Gateway
• Gateway Routing pattern - Azure Architecture Center
• The API gateway pattern versus the direct client-to-microservice communication - .NET
• What is Amazon API Gateway?
• Amazon API Gateway concepts - Amazon API Gateway

Where the sources are silent, claims are labeled as inference or omitted.

The Problem: Latency and Compliance in Global APIs

When a mobile app or a SaaS service talks to a backend that lives in one data center, every request has to cross the globe. The Azure Architecture Center notes that a single‑region endpoint can add 40–80 ms of round‑trip time for users on the opposite side of the world. That extra delay hurts user experience and can push a product below its SLA.

At the same time, regulations such as GDPR, CCPA, and India’s PDPB force data to stay within certain jurisdictions. If a request from a European user hits a U.S. backend, the provider violates the law. The only way to satisfy both latency and compliance is to route traffic locally while still keeping a single, centrally managed API surface.

Why It Matters: Scale, Latency, and Regulation

Modern applications serve millions of users spread across continents. The cost of a workload-dependent latency latency hit multiplies across every request, and the risk of a compliance breach can lead to fines and reputational damage. The stakes are high:

• Performance: Users expect sub‑workload-dependent latency responses; a 40–80 ms penalty can push a request over that threshold.
• Compliance: Data residency rules are non‑negotiable; a mis‑routed request can trigger legal penalties.
• Availability: A single point of failure in one region can bring down the entire service for users elsewhere.

Failure domains and blast radius

Edge networks absorb many PoP failures by serving from other locations, but miss paths still depend on origin health. Remaining risk concentrates in:

• Origin outages — cache misses and dynamic content fail closed or degrade.
• Misconfigured TTLs / cache keys — personalization bugs and stampedes.
• Edge compute errors — edge logic can fail before origin helps.
• Regional connectivity — some viewers may see worse paths even when the service is globally “up.”

Prefer vendor docs wording over invented PoP counts or latency SLAs.

The Core Insight: Edge‑First, Policy‑Aware Routing

The modern solution is to let the edge decide where a request should go, but to keep the routing logic in a central control plane. API gateways expose a single hostname, but the DNS resolution is Anycasted to the nearest edge POP. The edge node terminates TLS, applies routing rules (including geo‑policy), and forwards the request to the appropriate backend. This pattern keeps the surface simple for developers while satisfying latency, compliance, and failover requirements.

How API Gateways Route Requests: A 60‑Second Overview

User → DNS → Anycast Edge → Gateway Node → Routing Logic → Backend. (See the diagram below for a quick visual recap.)

Architecture and Topology

The gateway stack can be broken into three logical layers:

1. Ingress Layer – DNS, Anycast, TLS termination, and the first point of contact for the client.
2. Control Plane Layer – A globally‑distributed service that stores routing tables, policy bundles, and health‑check data.
3. Backend Layer – The actual microservice instances, often spread across multiple regions or cloud providers.

Each layer is independent, but they must coordinate tightly to keep the request path short. The ingress layer forwards the request to the control plane via a lightweight protocol (HTTP/2 or gRPC). The control plane then resolves the target backend and streams the decision back to the ingress node, which immediately forwards the request. This design keeps the per‑request latency to a single hop between ingress and control plane, and avoids a round‑trip to a central database for every request.

The control plane itself is a cluster of stateless nodes that run a consistent key‑value store (often a distributed database such as etcd or Consul). The store holds:

• Routing tables – mapping of API paths or hostnames to backend endpoints.
• Policy bundles – geo‑policy, rate‑limit, and authentication rules.
• Health‑check metadata – last‑known‑good status of each backend.

Because the store is replicated across regions, the control plane can serve requests from the nearest node, reducing the time to fetch a routing decision.

How a Request Traverses the System

1. DNS Lookup – The client resolves the gateway hostname. The DNS server is Anycasted; the nearest edge POP answers.
2. TLS Handshake – The edge node terminates TLS.
3. Policy Evaluation – The edge node forwards the request metadata (IP, headers, path) to the control plane.
4. Routing Decision – The control plane looks up the routing table, applies any geo‑policy, and returns the backend address.
5. Forwarding – The edge node forwards the request to the chosen backend over HTTP/2.
6. Response Path – The backend responds, the edge node streams the response back to the client, optionally applying caching or WAF rules along the way.

This flow is illustrated in the diagram below (mermaid is rendered by the pipeline).

Mechanism Deep Dive: Geo‑Policy Enforcement

Geo‑policy is the most visible feature of global routing. The policy is expressed as a set of rules that map IP ranges or country codes to a preferred set of backends. The control plane stores these rules in a compact, serialized format (often a protobuf or JSON blob). When a request arrives, the edge node performs a quick lookup:

• It extracts the client IP from the TLS handshake or the X-Forwarded-For header.
• It performs a binary search over a sorted list of CIDR blocks to find the matching country.
• It then consults the policy bundle to pick the backend with the lowest latency or highest capacity.

Because the lookup is purely in memory and the policy bundle is small (tens of kilobytes), the edge node can evaluate the policy in a few microseconds. If the policy changes, the control plane pushes a new bundle to all edge nodes via a publish/subscribe channel; the nodes atomically swap the old bundle for the new one.

Failure Modes

• Stale Policy – If a push fails, an edge node may continue using an outdated policy until the next successful push.
• IP Spoofing – Clients can spoof IPs; the gateway mitigates this by validating the X-Forwarded-For header against the TLS client IP.
• Backend Unavailability – If the chosen backend is down, the control plane marks it unhealthy and retries with an alternate backend.

Results and Tradeoffs

The public documentation does not publish exact latency numbers for the entire path, but it does highlight the benefits:

• Reduced Round‑Trips – By resolving routing decisions at the edge, the system eliminates a round‑trip to a central database for every request.
• Compliance – Geo‑policy ensures that data never leaves the region mandated by regulations such as GDPR.
• Scalability – The stateless ingress layer can scale horizontally without coordination, while the control plane remains a small, highly available cluster.

Tradeoffs include:

• Complexity – Maintaining a global control plane and pushing policy bundles adds operational overhead.
• Consistency – Policy updates are eventually consistent; a client may hit a different backend for a short period after a change.
• Cost – Running a global edge network and a distributed control plane incurs higher infrastructure costs compared to a single‑region deployment.

What I Would Steal

If I were building a small‑scale API gateway for a startup, I would adopt the following:

• Anycast DNS – Even a single edge node can benefit from Anycast to reduce the distance to the nearest client.
• Edge‑Based TLS Termination – Offloading TLS to the edge simplifies the backend and improves performance.
• Centralized Policy Store – Use a lightweight distributed KV store (etcd or Consul) for routing tables; it gives me consistency without a full‑blown control plane.
• Incremental Policy Pushes – Push policy changes as small JSON blobs; the edge nodes can swap them atomically, keeping the system responsive.

These ideas keep the architecture simple while still delivering the key benefits of global routing.

Frequently Asked Questions

Key Takeaways

• Global API routing is driven by latency, compliance, and availability needs.
• A three‑layer architecture—ingress, control plane, backend—keeps the system modular and scalable.
• Geo‑policy is enforced at the edge, with policies stored in a distributed KV store and pushed to edge nodes.
• The design trades off operational complexity for reduced latency and regulatory compliance.
• For small projects, adopting Anycast DNS, edge TLS termination, and a lightweight policy store captures most of the benefits without the full infrastructure overhead.

Architecture and Topology — Edge ↔ Control Plane ↔ Backend

The public docs describe a three‑layer topology:

1. Edge layer – Anycast‑advertised DNS names resolve to the nearest POP. Each POP runs a lightweight gateway process that terminates TLS, performs health‑checks, and forwards the request to the control plane for policy evaluation.
2. Control‑plane layer – A globally distributed key‑value store (e.g., Azure Cosmos DB, AWS DynamoDB Global Tables) holds routing policies, rate‑limit quotas, and compliance tags. Edge nodes pull a compact snapshot of the relevant policy set on start‑up and receive incremental updates via a publish‑subscribe channel.
3. Backend layer – One or more regional API services (Kubernetes clusters, serverless functions, or VM farms) host the actual business logic. The control plane returns a concrete target endpoint (region, version, or canary bucket) to the edge node, which then proxies the request.

This separation lets the edge stay stateless, the control plane stay consistent, and the backend scale independently.

How Requests Flow End‑to‑End

1. DNS resolution – The client resolves api.example.com. Anycast DNS returns the IP of the POP that is topologically nearest according to BGP routing. The docs note that this is routing‑nearest rather than strictly network-nearest.
2. TLS termination – The POP terminates TLS using a certificate that is either shared across POPs (via a global secret store) or provisioned per POP via an automated ACME workflow.
3. Ingress filtering – The edge node applies a static denylist (IP blocks, known bad user‑agents) before any further processing. This is documented as a “quick‑reject” path that avoids unnecessary control‑plane round‑trips.
4. Policy fetch – The node queries the control‑plane KV store for the request’s routing policy. The policy key is derived from the request’s host, path, and optional JWT claims. The store returns a JSON document that includes:

• Target region(s)
• Rate‑limit bucket identifiers
• Compliance tags (e.g., gdpr_eu)

1. Routing decision – The edge node selects a target region based on the policy and current health status (heartbeat pings from backend services). If the primary region is unhealthy, the node falls back to a secondary region as defined in the policy.
2. Proxy to backend – The request is proxied over an encrypted tunnel (TLS‑to‑TLS) to the chosen backend service. The edge does not cache responses unless the API explicitly enables edge caching via Cache‑Control headers; in that case the edge stores the response in an in‑memory LRU cache with a configurable TTL.
3. Response path – The backend returns the response, which traverses the same edge POP back to the client. If the response includes Set‑Cookie or other stateful headers, the edge forwards them unchanged.

Deep Dive: Distributed Policy Store

The control plane’s policy store is the heart of the routing logic. Public documentation highlights three design goals:

• Strong consistency for compliance – GDPR‑related tags must be visible to every POP at the same logical time. The store therefore uses a quorum‑based replication protocol (e.g., DynamoDB’s “strongly consistent reads”) rather than eventual consistency.
• Low‑latency reads at the edge – Edge POPs cache the policy locally and refresh it on a configurable interval (default ≈ 5 minutes) or when a version change is published. This reduces the per‑request latency to a single local memory lookup.
• Fine‑grained sharding – Policies are sharded by host name and path prefix, allowing a POP to fetch only the subset it needs. The docs show an example where a single tenant with 10 k APIs stores each API’s routing rule as a separate item, keeping the overall payload per fetch under a few hundred kilobytes.

{
 "policyId": "api.example.com/v1/orders",
 "targetRegions": ["us-east-1", "eu-west-1"],
 "fallbackRegion": "us-west-2",
 "rateLimit": {"bucket":"orders-api","limit":1000,"windowSec":60},
 "compliance": ["gdpr_eu"]
}

Operational notes from the docs

• Policy updates are published via a change‑stream that edge POPs subscribe to. The stream guarantees at‑least‑once delivery; POPs deduplicate based on the policy version field.
• If a POP loses connectivity to the control plane, it continues to serve traffic using the last‑known policy snapshot. The edge will reject requests that require a newer compliance tag, returning a several Forbidden with a “policy stale” message.
• The store does not expose raw read/write latency numbers; the documentation only says “reads complete in sub‑millisecond time for cached entries” and “writes propagate within seconds across regions.”

Results and Trade‑offs

Observed benefits (publicly reported)

• Latency reduction – By terminating TLS and performing routing decisions at the edge, the system avoids a full round‑trip to the control plane for every request. The docs state that “most requests complete with only one additional hop beyond the edge‑to‑origin path.”
• Compliance enforcement – Strongly consistent policy reads guarantee that data residency constraints are respected globally, eliminating the need for per‑region code branches.
• Operational isolation – Failures in one backend region do not cascade; the edge can instantly switch to a secondary region without redeploying edge software.

Known trade‑offs

• Increased edge complexity – The edge POP must run a lightweight policy engine, TLS termination, and health‑checking logic. This raises the operational surface area compared to a simple “edge‑only cache” model.
• Policy propagation latency – While the store aims for sub‑second propagation, the docs acknowledge a “few‑second window” where a newly issued compliance rule may not be enforced everywhere. During that window, a request could be routed to a non‑compliant region.
• Cache‑invalidation cost – If an API response is cached at the edge, invalidating it on a policy change requires a broadcast to all POPs, which can generate a burst of control‑plane traffic.

What I Would Steal

If I were building a startup API platform, I would adopt the edge‑first policy evaluation pattern:

• Anycast DNS + edge TLS termination – Gives me a “closest‑pop” experience without building a global load balancer from scratch.
• Lightweight policy cache – Store routing rules in a small KV store (e.g., Cloudflare Workers KV) and pull them on startup. The edge can then make routing decisions locally, keeping per‑request latency low.
• Fail‑over via health checks – Simple health‑check pings from each backend region allow the edge to switch targets on the fly, avoiding a full‑blown control plane for basic availability.

The biggest practical win is the separation of compliance logic from business code. By keeping residency tags in a central store, I can add a new GDPR‑only region without touching any API implementation.

Frequently Asked Questions

1. Does the edge ever contact the origin directly without consulting the control plane? Public docs state that the edge always performs a policy lookup first; there is no “origin‑only” shortcut.

2. How are TLS certificates managed across POPs? The documentation mentions automated certificate provisioning via ACME and optional secret replication; exact key‑management details are not public.

3. What happens if the policy store becomes unavailable? Edge POPs continue serving traffic using the last cached snapshot. Requests that require a newer compliance tag are rejected with a several.

4. Is the routing decision deterministic across POPs? Yes, because the policy document includes an explicit list of target regions and a deterministic fallback order.

5. Can I use a different KV store for the control plane? The architecture is abstracted to “a globally replicated KV store with strong consistency.” Any store that meets those guarantees (e.g., DynamoDB Global Tables, Cosmos DB) can be swapped in.

6. Do edge POPs cache API responses by default? Only when the API explicitly sets cache‑control headers. Otherwise the edge acts as a pure proxy.

7. How is rate‑limiting enforced? Rate‑limit buckets are part of the policy payload. The edge node increments a local counter and syncs it back to the control plane on a configurable interval.

8. Is there a built‑in DDoS mitigation layer? The docs reference a “quick‑reject” path for known bad IPs and a separate, undocumented DDoS protection service that sits in front of the Anycast DNS; details are not publicly disclosed.

How Cloudflare’s edge platform works: a 60‑second overview

When a client resolves a hostname that is proxied through Cloudflare, the DNS response points to an anycast‑advertised IP address. The client’s traffic is then routed by the Internet’s BGP system to the network‑nearest Cloudflare PoP—i.e., the PoP that can be reached with the fewest AS hops, not necessarily the network-nearest one.

At the edge PoP the request follows a deterministic pipeline:

1. Edge listener – a high‑performance TCP/UDP listener accepts the connection and hands it to the HTTP stack.
2. Routing & load‑balancing – Cloudflare’s internal routing layer selects the appropriate virtual hostname configuration (e.g., a specific zone, a load‑balancer pool, or a custom hostname).
3. Security filters – the request traverses the WAF, rate‑limit, bot‑management, and TLS termination stages. These filters are applied before any cache lookup, ensuring that malicious traffic never reaches downstream components.
4. Cache lookup – Cloudflare checks its edge cache for a fresh copy of the response. If a hit occurs, the cached payload is served directly to the client, bypassing the origin.
5. Worker execution (if configured) – a Cloudflare Worker script runs before the cache read when the route is bound to a Worker. The script can rewrite the request, generate a response, or decide to bypass the cache.
6. Origin fetch – on a cache miss (or when the Worker explicitly forwards), the edge node opens an outbound connection to the origin server, respecting any configured origin‑pull settings (e.g., TLS client certificates, custom headers).
7. Response processing – the origin response flows back through the same security filters, can be cached according to the zone’s cache‑control policy, and is finally streamed to the client.

All of these steps happen within a few milliseconds at the edge, and the path is the same regardless of whether the request ultimately hits the cache or the origin. The only variation is whether a Worker intervenes before the cache lookup, which is a documented configuration option in Cloudflare’s public docs.

Sources

• API gateways - Azure Architecture Center
• Create routes for HTTP APIs in API Gateway
• Gateway Routing pattern - Azure Architecture Center
• The API gateway pattern versus the direct client-to-microservice communication - .NET
• What is Amazon API Gateway?
• Amazon API Gateway concepts - Amazon API Gateway
• Deploy Oracle Database step by step on Amazon EVS with FSx for ONTAPZGateway: Learnings from Putting a Proxy in Front of ZippyDB](https://engineering.fb.com/2026/09/03/core-infra/zgateway-proxy-zippydb-meta/)

Image credits

• Cover: AI-generated illustration