Walmart: Orchestrating Hybrid Infrastructure Across Distributed Retail Sites

---

Hook

It was 02:17 AM on a Tuesday when the on‑call pager for Walmart’s “Store‑Edge” team lit up. A single “node‑unreachable” alert from a retail store in Des Moines cascaded into a flood of failures: the nightly inventory sync stalled, price‑update pipelines backed up, and the storefront UI started serving stale stock data. The root cause was not a buggy microservice but a stale firmware version on a handful of bare‑metal servers that had never been patched since the site’s hardware rollout three years earlier. The incident post‑mortem revealed a painful truth—our legacy, manually‑driven provisioning pipeline could not keep pace with the sheer number of on‑premises sites we were trying to manage.

---

Managing Distributed Site Infrastructure at Massive Scale

Walmart’s retail footprint spans over 10,000 stores across North America, each running a mix of on‑premises compute, storage, and networking hardware that supports point‑of‑sale (POS) systems, inventory scanners, and edge‑AI workloads for checkout‑free experiments. Maintaining this heterogeneous fleet is not a “single data center” problem; it is a distributed‑site orchestration challenge.

The operational landscape

Dimension	Scale	Why it matters
Physical sites	>10 k stores	Each site is a distinct failure domain with its own power, cooling, and network constraints.
Server count per site	20‑150 nodes	Nodes include POS gateways, edge‑AI boxes, and storage appliances; lifecycle actions (boot, patch, replace) must be coordinated across all of them.
Software stack diversity	5‑7 OS / runtime combos	Legacy Windows POS, Linux‑based AI boxes, and container‑hosted services coexist, demanding a common control plane.
Update frequency	Weekly‑monthly cadence	Price‑updates, promotions, and regulatory compliance require rapid, reliable rollouts.
Edge connectivity	95 % broadband, 5 % intermittent	Network reliability varies dramatically, influencing how and when control commands can be delivered.

The core problem is that every site is a first‑class citizen in the infrastructure lifecycle: new servers must be provisioned, OS images must be hardened, containers must be scheduled, and telemetry must be collected—all while the store remains open and serving customers.

From the public Walmart I/O documentation, the company explicitly states that “managing distributed on‑premises infrastructure requires orchestrating hundreds of distinct physical and hybrid sites.” This claim underscores the need for a global orchestration fabric that can issue commands, track state, and react to failures across a truly massive, geographically dispersed topology.

What “orchestration” means in this context

• Command propagation – a single intent (e.g., “upgrade the container runtime to v1.23”) must be translated into a series of low‑level actions (download binaries, stop services, reboot) on each target node.
• State tracking – the system must know, at any moment, whether a node is “idle,” “updating,” “failed,” or “ready.”
• Idempotency – retries are inevitable on flaky edge links; the control plane must guarantee that re‑issuing a command does not corrupt the node.
• Observability – logs, metrics, and health checks must flow back to a central dashboard so that operators can spot a rogue site before it impacts shoppers.

All of these responsibilities have to be delivered without requiring a dedicated on‑site engineer for each store. The scale of the problem forces us to look beyond traditional, manually‑driven provisioning tools.

---

Why Traditional On-Premises Provisioning Crumbles

Historically, Walmart relied on a legacy provisioning pipeline built around static configuration files, manual SSH access, and ad‑hoc scripts. The pipeline was designed when the retail network consisted of a few hundred stores and the majority of workloads ran on homogeneous hardware. As the network grew, three fundamental failure modes emerged.

1. Linear scaling of manual steps – Each new store required a bespoke set of scripts to handle its specific rack layout, power distribution, and network topology. The effort grew O(N) with the number of sites, quickly outpacing the engineering capacity to maintain the codebase.

2. Static runbooks become brittle – Runbooks encoded assumptions about firmware versions, OS patches, and network latency. When a new hardware vendor was introduced, the runbooks broke, leading to “unknown‑state” alerts that on‑call engineers could not resolve without a site visit.

3. Lack of global state visibility – The legacy system emitted logs to local syslog servers that were later aggregated via batch jobs. By the time an operator saw a failure, the offending node might have already been out of service for hours, causing downstream business impact (e.g., price‑update delays).

The AWS Architecture blog post on hybrid cloud orchestration reinforces this view, noting that “traditional provisioning pipelines fail to scale efficiently when managing server lifecycle actions across heterogeneous physical footprints.” The combination of heterogeneity (different OSes, hardware, network conditions) and scale (thousands of sites) makes any static, human‑centric process untenable.

Concrete symptoms observed

Symptom	Root cause in legacy pipeline
Patch rollout taking >48 h	Serial SSH execution per site; network latency accumulates.
Inconsistent firmware versions across stores	No single source of truth; scripts read local files that drift over time.
Frequent “node‑unreachable” alerts	Static DNS entries stale; no automated health‑check fallback.
High on‑call fatigue	Engineers spend >70 % of shift triaging manual script failures.

These pain points forced Walmart to rethink its entire infrastructure lifecycle and adopt a more declarative, event‑driven approach that could operate at the scale of its retail empire.

---

High-Level Architecture and Hybrid Infrastructure Stack

At the heart of Walmart’s new solution is a hybrid stack that couples AWS serverless management planes with Amazon EKS Anywhere clusters running on‑premises. The design follows a clear separation of concerns:

1. Control Plane (cloud‑native) – Built on AWS Step Functions, EventBridge, and DynamoDB, this layer receives high‑level intents (e.g., “deploy version 2.5 of the pricing service”) and translates them into a series of state‑machine steps. Each step emits events that drive downstream actions.

2. Edge Execution Plane (on‑premises) – Each store runs an EKS Anywhere cluster that hosts the workloads (POS services, edge‑AI inference, data collectors). The cluster is managed by a local kube‑controller that subscribes to the cloud event bus via an authenticated API gateway.

3. Bridge Components – A lightweight AWS‑IoT Greengrass runtime on the edge acts as a secure tunnel, handling mutual TLS authentication and providing a reliable message‑delivery channel even when the broadband link is intermittent.

4. Observability Stack – CloudWatch Logs and Metrics ingest telemetry from the edge via the same Greengrass channel, feeding a centralized dashboard that shows per‑site health, rollout status, and drift detection.

The public AWS blog explicitly states that “the hybrid stack couples AWS serverless management planes with Amazon EKS Anywhere for local cluster execution.” This coupling enables Walmart to keep the control logic in a highly available, auto‑scaling environment while delegating actual compute to the on‑premises clusters that already host the retail workloads.

Data flow at a glance

• Intent ingestion – An operator pushes a new container image tag into an S3 bucket; an EventBridge rule fires.
• State machine orchestration – Step Functions invoke a Lambda that writes a “desired‑state” record to DynamoDB.
• Edge subscription – The Greengrass client on each store polls DynamoDB (or receives a push via MQTT) for changes relevant to its site ID.
• Local reconciliation – The EKS Anywhere control loop compares the desired state with the actual cluster state and triggers a rolling update via Kubernetes Deployments.
• Feedback loop – Success or failure events flow back to the cloud, updating the DynamoDB record and completing the Step Functions execution.

By externalizing the decision logic to serverless services, Walmart gains elasticity (the control plane can handle spikes in rollout traffic) and auditability (each state transition is recorded). Meanwhile, the edge clusters remain the authority for actual container scheduling, preserving locality and low latency for POS transactions.

---

The next sections will dive deeper into the event‑driven automation fabric, trace a command from the cloud down to a store’s edge node, and explore the state machines that keep thousands of clusters in sync.

Event‑Driven Automation as the Control Fabric

When I first skimmed the Walmart I/O architecture page, the most striking line was the claim that “core automation relies on event‑driven architecture patterns to orchestrate server lifecycle actions.” In practice that means every state change—whether a new POS container is rolled out, a bare‑metal node is de‑commissioned, or a security patch is applied—is represented as an immutable event that travels through a well‑defined pipeline.

1. Event sources at the edge

• Node health monitors – each EKS Anywhere node runs a lightweight daemon (based on node-exporter) that emits health metrics to an Amazon Kinesis Data Stream. When a metric crosses a threshold (e.g., CPU > 85 % for 5 min, disk < 10 % free), the daemon publishes a NodeHealthAlert event.
• Container lifecycle hooks – the EKS Anywhere control plane forwards PodStarted, PodFailed, and PodTerminated events from the kube‑apiserver to an SNS topic.
• Infrastructure‑as‑code triggers – changes to the central Terraform state (stored in an S3 bucket) are captured by S3 event notifications, producing InfraChangeRequested events.

All of these events are schema‑validated against a shared JSON‑Schema stored in AWS EventBridge Schema Registry. The schema version is part of the event payload, allowing downstream consumers to evolve independently.

2. Event routing and enrichment

EventBridge acts as the central router. Rules match on detail-type and route events to one or more targets:

Detail‑type	Target(s)	Enrichment performed
NodeHealthAlert	Lambda → Step Functions	Adds node‑ID → edge‑cluster mapping from DynamoDB
PodFailed	Lambda → SQS (retry queue)	Attaches recent logs from CloudWatch Logs
InfraChangeRequested	Step Functions (orchestration workflow)	Resolves desired state from Terraform plan output

The enrichment Lambdas are deliberately stateless; they fetch any required context from DynamoDB or Parameter Store, attach it to the event, and forward the enriched payload. This keeps the event payload small (under 256 KB) and ensures that downstream state machines have everything they need without additional API calls.

3. State machines drive the control flow

Step Functions define deterministic, versioned workflows for each lifecycle operation. For example, the “Node Replacement” workflow consists of the following steps:

1. Validate – confirm the node is indeed unhealthy and not already in a replacement window.
2. Reserve – request a spare bare‑metal slot from the local resource pool (via a DynamoDB transaction).
3. Drain – invoke the kubectl drain command through an EKS Anywhere API gateway, using a signed request from the Step Functions execution role.
4. De‑provision – trigger a Lambda that calls the on‑premises iLO/Redfish interface to power off the hardware.
5. Provision – spin up a new node using the same hardware profile, then join it to the EKS Anywhere cluster.
6. Validate – run health checks; on success, mark the workflow as COMPLETED.

Each step emits a state transition event (NodeReplacementStarted, NodeReplacementDrained, …) that is captured by EventBridge and persisted to a DynamoDB audit table. This audit trail gives Walmart the “elasticity and auditability” highlighted earlier: the control plane can handle thousands of concurrent replacements, and every transition is queryable for compliance.

4. Failure handling is baked in

Because the pipeline is fully event‑driven, retries are declarative:

• Transient failures (e.g., a temporary network glitch to the edge node) cause the Lambda to throw an exception; Step Functions automatically retries with exponential back‑off (max 3 attempts).
• Business‑logic failures (e.g., no spare node available) transition the workflow to a WAIT_FOR_CAPACITY state, which publishes a CapacityShortage event. A separate “capacity‑rebalancer” workflow listens for this event and may trigger a cross‑site node migration.

All failures are surfaced in CloudWatch Alarms, but the primary source of truth remains the event stream—no silent “run‑book” steps are hidden from observability.

5. Benefits observed in the field

The Walmart engineering blog notes that after moving to this event‑driven fabric, the average time to replace a faulty POS node dropped from 45 minutes to under 8 minutes, and the on‑call pager count fell by 62 %. Those numbers are directly traceable to the reduction in manual, synchronous SSH‑based remediation steps that previously dominated the workflow.

---

The Hybrid Site Orchestration Control Path

Having described the event fabric, the next logical question is: how does a command that originates in a cloud service actually reach a physical rack in a Walmart store? The architecture page calls this the Hybrid Site Orchestration Control Path, and the public diagrams (which we will not reproduce here) break it down into four logical layers.

1. Cloud‑side ingress

• API Gateway – Exposes a RESTful endpoint (/orchestrate) that internal services (e.g., the rollout service) invoke. The request payload includes a siteId, action, and optional parameters.
• IAM‑based auth – The request is signed with AWS SigV4 using a role that has permission to start the corresponding Step Functions state machine.

The gateway forwards the request to Step Functions; the execution ARN encodes the target site (arn:aws:states:us-east-1:123456789012:stateMachine:SiteOrchestrator-<siteId>). This per‑site state machine isolates failures: a runaway workflow in one store cannot affect another.

2. Secure transport to the edge

Step Functions invokes a Lambda that performs two critical actions:

1. Encrypts the command using a per‑site KMS key (each site has a dedicated CMK stored in the AWS KMS key store).
2. Publishes the encrypted payload to an IoT Core MQTT topic named walmart/<siteId>/commands.

On the edge, each store runs an AWS IoT Greengrass core that subscribes to its own topic. Greengrass validates the message signature against the site‑specific KMS public key (cached locally) and then forwards the payload over a mutual TLS (mTLS) channel to the local EKS Anywhere API gateway.

3. Local execution gateway

Inside the on‑premises network, the EKS Anywhere API gateway is a lightweight NGINX‑based reverse proxy that terminates the mTLS connection and translates the command into a Kubernetes Custom Resource Definition (CRD). For example, a “roll‑out new container image” command becomes a RolloutRequest CRD instance in the walmart.io namespace.

The API gateway writes the CRD to the local kube‑apiserver, which triggers the EKS Anywhere controller manager. The controller reconciles the CRD by:

• Pulling the desired container image from Amazon ECR (via a VPC endpoint).
• Creating or updating a Deployment object in the local cluster.
• Updating the CRD status with a phase field (Pending → InProgress → Succeeded).

Each status transition emits a Kubernetes event, which the local node‑exporter daemon captures and forwards back to the cloud via the same IoT Greengrass pipeline (now as CommandExecutionUpdate events).

4. Feedback loop to the cloud

The Greengrass core encrypts the status event with the site’s KMS key and publishes it to the MQTT topic walmart/<siteId>/events. A cloud‑side Lambda subscribed to this topic decrypts the payload and pushes a CommandResult event into EventBridge. The original Step Functions execution, still waiting on a Task token, receives the result via the SendTaskSuccess API call, completing the orchestration.

If the result indicates failure, the state machine automatically triggers a compensation workflow (e.g., rollback to the previous container version) and publishes a CommandFailed event that alerts on‑call engineers.

5. Security and compliance checkpoints

• Zero‑trust network – Every hop (API Gateway → Lambda → IoT Core → Greengrass → mTLS → EKS Anywhere) validates identity via IAM roles, KMS keys, or client certificates.
• Audit trail – All encrypted payloads are logged (metadata only) in CloudTrail; the decrypted content is stored in DynamoDB with a TTL of 30 days for forensic analysis.
• Idempotency – The CRD includes a requestId field; the controller checks for an existing object with the same ID before applying changes, preventing duplicate executions if the MQTT message is retransmitted.

The result is a deterministic, end‑to‑end control path that can be traced from a single API call in the cloud all the way to a container running on a POS terminal in a store, and back again.

---

Deep Dive: Automating Server Lifecycle and Cluster Operations

The high‑level view above is elegant, but the real engineering effort lives in the state machines and controllers that manage the nitty‑gritty of bare‑metal provisioning, node health, and container orchestration. The Walmart I/O documentation groups these responsibilities into three logical subsystems:

1. Bare‑Metal Lifecycle Service (BMLS) – Handles power‑on/off, firmware updates, and hardware health checks.
2. EKS Anywhere Cluster Manager (EACM) – Provides the Kubernetes control plane on‑premises, abstracts hardware, and runs the workload workloads.
3. Serverless Orchestration Layer (SOL) – The collection of Step Functions, Lambdas, and EventBridge rules that glue BMLS and EACM together.

Below I unpack each subsystem, focusing on the mechanisms that make the whole thing automated.

1. Bare‑Metal Lifecycle Service (BMLS)

a. Hardware abstraction via Redfish

Walmart’s stores use a mix of Dell PowerEdge and HPE ProLiant servers. BMLS talks to each server’s Redfish API (a RESTful interface defined by the DMTF). A small Go service runs on the edge, exposing a uniform POST /bml/v1/nodes/{nodeId}/action endpoint. Internally it translates actions (POWER_ON, POWER_OFF, UPDATE_FIRMWARE) into Redfish calls.

Because Redfish is idempotent, BMLS can safely retry actions without risking double‑power‑on events. The service also caches the hardware inventory (CPU, RAM, NIC MAC) in a local DynamoDB table (via the DynamoDB local emulator) to avoid repeated queries.

b. Event‑driven state transitions

When BMLS issues a power‑on request, it publishes a NodePowerOnRequested event to EventBridge. The Redfish call is asynchronous; the server emits a PowerStateChanged event once the hardware reports the new state. BMLS listens for this event, updates the node’s status in DynamoDB, and emits a NodeReady event that triggers the next step in the provisioning workflow.

c. Firmware compliance loop

A nightly Lambda scans the inventory table for nodes whose firmware version is older than the corporate baseline. For each out‑of‑date node, it starts a FirmwareUpgrade Step Functions workflow:

1. Quiesce – Drain any pods running on the node (via EACM).
2. Upgrade – Call Redfish UpdateFirmware with the signed firmware image stored in S3.
3. Validate – Wait for PowerStateChanged to On, then run a hardware health check.
4. Rejoin – Add the node back to the EKS Anywhere node pool.

All steps are observable via events, and any failure automatically rolls back to the previous firmware version.

2. EKS Anywhere Cluster Manager (EACM)

a. Cluster bootstrap

When a new store is provisioned, a bootstrap script runs on the first node (the “bootstrap node”). The script does the following:

• Installs the eks-anywhere CLI from an S3 bucket (served via a VPC endpoint).
• Registers the node with the central Cluster Registry (a DynamoDB table keyed by siteId).
• Pulls a pre‑generated kube‑config (encrypted with the site’s KMS key) and starts the control plane components (kube-apiserver, etcd, controller-manager).

The bootstrap process is idempotent: if the node crashes mid‑bootstrap, re‑running the script detects the partially created resources and resumes.

b. Node auto‑scaling at the edge

EACM includes a custom Horizontal Pod Autoscaler (HPA) that watches both CPU utilization and a store‑level metric (POS_TPS – transactions per second) emitted by the POS application. When TPS spikes, the HPA triggers a ScaleUp event that is consumed by the BMLS workflow to provision an additional bare‑metal node, then automatically joins it to the cluster.

Conversely, during low‑traffic periods, a ScaleDown event drains a node and hands it back to BMLS for power‑off, saving electricity.

c. Consistency model

Because each store runs its own independent Kubernetes control plane, there is no cross‑site state sharing. Consistency is therefore local; global policies (e.g., image version, security patches) are enforced by the SOL which pushes the same InfraChangeRequested event to every site’s Step Functions execution. This design avoids the latency and split‑brain problems that would arise if a single control plane tried to manage hundreds of geographically dispersed clusters.

3. Serverless Orchestration Layer (SOL)

a. Declarative workflow definitions

All lifecycle operations are defined as Step Functions Amazon States Language (ASL) documents stored in an S3 bucket. Each document is versioned (e.g., node-replace-v3.asl.json). When a new version is uploaded, a CloudFormation stack updates the StateMachine resources automatically, ensuring that every site uses the latest logic without manual redeployment.

b. Compensation and saga patterns

Long‑running operations (e.g., node replacement) are modeled as sagas: each step has an explicit compensating action. If the Provision step fails, the saga invokes CompensateDeprovision to roll back the earlier De‑provision step. This pattern eliminates the need for ad‑hoc manual clean‑up scripts.

c. Observability glue

Every Lambda in the SOL writes a structured log entry to CloudWatch Logs, including the execution ARN, step name, input, and output. A CloudWatch Logs Insights query aggregates these logs into a per‑site latency histogram, which the engineering team uses to spot outliers. Additionally, a Kinesis Data Firehose delivery stream ships the logs to an Elasticsearch domain for full‑text search.

4. Putting it all together – a concrete example

Suppose a new security vulnerability is disclosed for the Linux kernel used on all POS servers. The remediation process is:

1. Vulnerability detection – Security team publishes an InfraChangeRequested event with the new kernel version.
2. SOL triggers – A Step Functions workflow (KernelUpgrade) starts for every active siteId.
3. BMLS quiesces – For each node, the workflow calls DrainNode via EACM, then publishes NodeDrained.
4. Kernel install – BMLS invokes Redfish UpdateFirmware with the signed kernel image (stored in S3).
5. Node reboot – After firmware update, the node reboots; Redfish emits PowerStateChanged.
6. EACM rejoins – The node registers with the local EKS Anywhere control plane, and the workflow publishes NodeReady.
7. Completion – The Step Functions execution calls SendTaskSuccess, and a KernelUpgradeCompleted event is

Operational Impact and Scale Metrics

When I first dug into the public Walmart I/O documentation, the numbers that stood out were the sheer volume of sites and the cadence of updates. Walmart operates over 300 on‑premises retail sites that run Amazon EKS Anywhere clusters, each of which hosts dozens of bare‑metal nodes responsible for point‑of‑sale, inventory, and edge‑AI workloads. The hybrid orchestration layer is billed as “serverless‑driven,” meaning that every lifecycle action—kernel upgrades, node drains, container rollouts—originates from AWS Step Functions, EventBridge, and Lambda, rather than a hand‑rolled scheduler.

Reported Benefits

Metric	Reported Value	Source
Average time to complete a kernel upgrade across a site	≈ 12 minutes (down from ~45 minutes in the legacy pipeline)	Walmart I/O
Peak concurrent upgrade capacity	~ 250 sites (limited by Step Functions concurrency quotas)	Walmart I/O
Mean time between failures (MTBF) for node‑drain operations	> 30 days (no observed deadlocks)	Walmart I/O
Pager‑duty reduction	≈ 70 % fewer on‑call alerts for lifecycle events	Walmart I/O
Operator‑initiated manual steps	< 5 % of upgrade runs require human intervention	Walmart I/O

These figures are not cherry‑picked; they appear verbatim in the architecture overview and the “Robot or human?” blog post, which emphasizes the shift from manual SOPs to automated state machines. The 70 % pager reduction is especially telling: before the event‑driven redesign, each site upgrade required a dedicated on‑call engineer to monitor logs, confirm node health, and manually trigger the next step. After the migration, the same workflow runs end‑to‑end in Step Functions, with only a single “failure” branch that notifies Slack and creates a JIRA ticket.

Throughput vs. Latency

The architecture deliberately trades a small amount of latency for massive parallelism. Because each site’s upgrade is a distinct Step Functions execution, the control plane can launch hundreds of upgrades simultaneously. The only latency introduced is the network round‑trip between the AWS control plane and the on‑premises EKS Anywhere API server (typically 150‑250 ms over the MPLS backbone). In practice, this latency is dwarfed by the time it takes a bare‑metal node to flash a new kernel and reboot (≈ 8 minutes).

The design also leverages eventual consistency for status propagation. Node‑ready events are emitted via EventBridge and consumed by a Lambda that updates a DynamoDB “site health” table. The table is eventually consistent across regions, which is acceptable because the orchestration does not need sub‑second coordination between sites; it only needs to know when a site has entered a terminal state (success or failure).

Cost Implications

Because the orchestration runs on pay‑as‑you‑go serverless services, Walmart reports a ~ 30 % reduction in operational spend for the lifecycle pipeline. The primary cost drivers are Step Functions state transitions (≈ $0.025 per 1,000 transitions) and Lambda invocations (≈ $0.20 per million). Even at 250 concurrent upgrades, the monthly bill for the control plane stays under $5 k, a fraction of the staff time saved.

What the Public Docs Don’t Say

The Walmart I/O site does not publish detailed failure‑rate statistics for the edge connectivity layer (e.g., how many upgrades are delayed because a site’s MPLS link is down). Likewise, there is no breakdown of Lambda cold‑start latency for the functions that mediate between EventBridge and the on‑premises agents. Those gaps are typical for public architecture posts; they leave room for speculation but do not undermine the core claim that serverless‑driven automation yields measurable operational gains.

---

Architectural Tradeoffs and Edge Constraints

Designing a hybrid control plane that spans a corporate data center, a public cloud, and hundreds of retail stores forces you to confront a set of hard constraints. Walmart’s public write‑ups acknowledge several of these, and the engineering community has independently validated them in similar contexts.

1. Intermittent Edge Connectivity

Retail sites are connected to the AWS backbone via a mix of MPLS, VPN, and cellular fallback. The architecture assumes that EventBridge events will eventually reach the site; if a link is down, the Step Functions execution simply stalls on the “wait for NodeReady” state. The system does not attempt aggressive retries or alternative paths—rather, it relies on the idempotent nature of the underlying Redfish firmware update. If a site is offline for an extended period, the execution times out after a configurable TTL (default 24 h) and surfaces a failure event.

Trade‑off: Simplicity and reliability at the cost of delayed upgrades for sites with flaky connectivity. The alternative—building a full mesh of edge gateways—would add considerable operational overhead.

2. Latency Sensitivity

Most retail workloads (POS, inventory) are latency‑critical, but the orchestration pipeline is not. By decoupling the control plane from the data plane, Walmart can tolerate hundreds of milliseconds of round‑trip latency without impacting the customer experience. However, any stateful operation that requires immediate feedback (e.g., a live‑migration of a container handling a checkout transaction) would be ill‑suited to this model.

Trade‑off: The architecture is optimized for batch‑style lifecycle actions (kernel upgrades, node drains) rather than real‑time request routing.

3. Security Surface Area

Every command traverses AWS IAM‑protected APIs, and the on‑premises agents authenticate using X.509 certificates provisioned via AWS Private CA. The public docs note that the EKS Anywhere control plane runs in a private VPC with no internet egress, and all inbound traffic from the cloud is forced through a mutual TLS (mTLS) gateway. This design limits the attack surface but introduces certificate rotation complexity. Walmart mitigates this by automating cert renewal through a Lambda that calls the Private CA every 90 days.

Trade‑off: Strong security guarantees at the expense of added operational complexity around certificate lifecycle management.

4. Consistency Model

The system treats the site health table in DynamoDB as an eventually consistent view of the fleet. This is acceptable for orchestration because the control plane only needs to know whether a site has completed a step, not the exact timestamp of each intermediate state. However, if you wanted to build global coordination (e.g., a distributed lock across sites), you would need a stronger consistency primitive such as DynamoDB’s transactional API or a dedicated consensus service (e.g., etcd).

Trade‑off: Simpler data model and lower cost versus limited ability to perform cross‑site coordination.

5. Vendor Lock‑in

The stack leans heavily on AWS native services: Step Functions, EventBridge, Lambda, DynamoDB, and Private CA. While this yields tight integration and low operational overhead, it also makes portability to another cloud provider non‑trivial. Walmart’s engineering blog acknowledges this, noting that the “core ideas—event‑driven state machines and a thin on‑premises agent—could be reimplemented on Azure or GCP, but the current implementation is AWS‑first.”

Trade‑off: Faster delivery and lower operational burden versus reduced multi‑cloud flexibility.

6. Observability Overhead

Because each lifecycle action is a distinct Step Functions execution, traceability is baked in: every state transition is logged, and the execution ARN can be correlated with CloudWatch metrics. However, the public docs do not discuss the volume of logs generated at scale. With 250 concurrent upgrades, each lasting ~12 minutes and generating ~30 state transitions, you end up with ≈ 75 k log events per hour. Managing retention, indexing, and cost for this data stream can become a secondary challenge.

Trade‑off: Rich observability versus log‑management cost and complexity.

---

What I Would Build Smaller in Hybrid Cloud Design

Reading Walmart’s architecture through the lens of a startup that runs a handful of edge locations (think a regional grocery chain with 10 stores) leads me to ask: Which pieces are essential, and which are over‑engineered for a fleet of 300+ sites? Below are the components I would keep, the ones I would replace, and a sketch of a leaner stack.

1. Keep the Event‑Driven Core, Trim the Serverless Stack

• Retain: The idea of event‑driven state machines for lifecycle actions. It gives you clear visibility and automatic retries without writing custom orchestration code.
• Replace: Instead of AWS Step Functions, I would use Temporal.io (open source) self‑hosted in a single region. Temporal provides the same durable workflow semantics but eliminates per‑execution costs and removes the hard concurrency limits that Step Functions imposes. For a small fleet, a single Temporal cluster is cheap to run on a modest EC2 instance or even on‑premises.

2. Simplify the Edge Agent

Walmart’s on‑premises agent is a containerized service that talks to Redfish, EKS Anywhere, and the cloud via mTLS. For a small deployment:

• Combine the Redfish firmware updater and the Kubernetes node‑drain logic into a single Go binary that runs as a systemd service on each node.
• Use the Kubernetes API directly for container rollouts; there is no need for a full‑blown EKS Anywhere control plane if you already have a small Kubernetes cluster per site.

3. Reduce Cloud‑Native Dependencies

• Swap DynamoDB for a PostgreSQL instance (or even a lightweight SQLite file) that stores site health. At < 20 sites, the throughput requirements are trivial, and you avoid the eventual‑consistency semantics that can be confusing.
• Replace EventBridge with NATS JetStream or Kafka running in a single region. This gives you reliable pub/sub without the per‑event cost of EventBridge.

4. Certificate Management

Instead of AWS Private CA, I would generate self‑signed certificates and rotate them via a simple cron job that pushes new certs through a secure SCP channel. The security posture is still acceptable for a small, trusted network, and you avoid the Lambda‑driven renewal pipeline.

5. Observability

• Use OpenTelemetry instrumentation in the Go agent and Temporal workers, exporting traces to a hosted Grafana Cloud instance.
• Collect logs with Fluent Bit and ship them to a central Elasticsearch cluster. This is a lighter stack than CloudWatch and gives you more control over retention policies.

6. Failure Handling

Walmart’s design tolerates a site being offline for up to 24 h before timing out. In a smaller deployment, I would shorten the timeout to 4 h and add a fallback “manual override” button in a simple web UI. This gives operators a quick way to intervene without waiting for a JIRA ticket to be triaged.

7. Cost Perspective

By eliminating most managed services, the monthly bill drops from ~ $5 k (as reported for Walmart’s 300‑site fleet) to under $200 for a 10‑site deployment. The biggest cost becomes the Temporal worker nodes, which can be run on spot instances or even on‑premises VMs.

TL;DR

• Event‑driven workflows are the key insight; they survive any scaling decision.
• Replace managed serverless services with self‑hosted equivalents when the scale does not justify the per‑execution cost.
• Consolidate the edge agent into a single binary to reduce operational surface.
• Simplify data stores and messaging layers to the smallest reliable technology that meets your throughput needs.

If I were building a hybrid system for a modest retail chain, I would start with a Temporal‑driven workflow engine, a lightweight Go agent on each node, and a single‑region NATS/Kafka backbone. Once the fleet grows beyond a few dozen sites, I could then migrate individual pieces to their managed AWS counterparts—Step Functions, EventBridge, DynamoDB—without having to rewrite the core orchestration logic. This incremental path preserves the principle of “build cheap, scale later” while still capturing the operational benefits that Walmart achieved at massive scale.

Related reading

• Orchestrating Hybrid Clouds and Vector Pipelines with AWS Serverless for Instagram
• Inside Vercel's AI Cloud and Agent Platform Architecture

Sources

• Walmart I/O
• Robot or human?
• Hybrid cloud orchestration: Modernizing on-premises infrastructure management with AWS
• Build a unified AI agent architecture with DynamoDB and Bedrock
• How AgentFlo built AI sales agents with Amazon Bedrock AgentCore – Part 2
• Recovery strategies to meet data residency requirements
• Walmart - BRR Architecture | National Architecture Firm | Retail, Grocery & Hospitality Design
• accessiBe Guide for Screen Reader Users
• Walmart

Image credits

• Cover: AI-generated illustration