Inside Pinterest Architecture: Scaling Real-Time Recommendations and Cloud-Native PubSub

Hook

I was scrolling through a public incident post on Pinterest’s engineering blog when the on‑call pager lit up at 02:13 AM UTC. The alert read: “MemQ write latency > 500 ms on Topic user‑actions – downstream recommendation pipeline stalled.” The accompanying screenshot showed a heat map of broker CPU spikes and a cascade of timeouts in the real‑time recommendation service. Within minutes the team rolled a new storage tier into the MemQ cluster, the latency dropped back below 50 ms, and the recommendation pipeline resumed serving pins. The incident was framed as a “traffic‑growth rebalancing” problem – a classic symptom of a tightly coupled PubSub stack that can’t absorb a sudden surge without costly manual sharding.

Stakes

Pinterest reports hundreds of millions of active users who generate billions of pin‑view, save, and click events each day. Those events feed a recommendation engine that must surface personalized content within a few hundred milliseconds of a user opening the app. The scale is not just in request volume; it’s also in geographic spread (data centers in North America, Europe, and APAC) and in the compute budget required to run large Transformer models for real‑time ranking. A single millisecond of added latency can translate to measurable drops in engagement, and the underlying PubSub layer is the first hop for every user action.

Why the obvious design breaks

1. Expensive rebalancing – Traditional Kafka clusters require a full stop‑the‑world rebalance whenever a new broker is added or a partition hot‑spreads. The post‑mortem notes that a 30 % traffic increase forced a week‑long rebalance that ate up engineering time and cloud spend.
2. Coupled storage and serving – In the legacy stack, the broker process also owned the log segment files. Scaling reads independently of writes meant over‑provisioning disk I/O on every broker, inflating costs.
3. Static runbooks – The Kafka‑centric runbooks assumed a fixed number of partitions per topic. When a new product feature doubled the fan‑out factor, the static scripts could not keep up, leading to the latency spike that triggered the pager.

These failure modes are exactly what Pinterest’s engineers call the “Kafka ceiling” in their public talks.

Reframe

The core insight that emerged from the MemQ project is simple: decouple the durable storage of the event log from the serving brokers, and make the storage layer pluggable. By treating the log as an object‑store‑backed append‑only ledger (similar to Pulsar or Facebook LogDevice) and letting brokers focus solely on request routing and metadata, Pinterest can scale writes and reads independently, replace storage back‑ends without code changes, and eliminate the heavyweight partition rebalancing that plagued their Kafka deployment.

Architecture overview

At a high level the new stack consists of four logical layers:

1. MemQ Client – SDK embedded in Pinterest services that discovers the cluster via a seed node, fetches topic metadata, and opens a TCP connection to the appropriate broker.
2. MemQ Broker – Stateless front‑end that hosts TopicProcessors, maintains in‑memory cursors, and forwards write requests to the storage layer while serving reads from the replicated log.
3. Pluggable Replicated Storage Layer – An interchangeable backend (Object Store, Distributed File System, or NFS) that stores the immutable append‑only log segments. Replication is handled by the storage system, not the broker.
4. TransAct – A PyTorch 1.12 module (Python 3.9.7+) that consumes the real‑time stream of user actions, runs a Transformer model, and emits ranked pin recommendations back into the feed pipeline.

The diagram that will be inserted under High‑Level Architecture of MemQ and TransAct shows these components and the data flow between them.

How it works

When a Pinterest service (e.g., the mobile feed backend) wants to record a user click, the MemQ client contacts a well‑known seed node to obtain the current cluster map. The seed node returns the address of the metadata broker that holds the TopicProcessor for user‑actions. The client then opens a persistent connection to that broker and streams the event. The broker writes the payload to the pluggable storage backend, which immediately replicates the segment across three zones. For consumers, the same broker reads from the replicated log, applies any required filtering, and pushes the record to the TransAct inference service. Because reads and writes hit separate paths, the system can sustain GB/s traffic on the write side while scaling read workers independently.

Deep dive

The next sections will unpack each piece:

• Core Mechanism: Decoupled Storage and Pluggable Replicated Layers – how MemQ abstracts the storage API and swaps S3, GCS, or an on‑prem DFS without redeploying brokers.
• The Read and Write Data Path in MemQ – a step‑by‑step trace of client discovery, metadata lookup, broker interaction, and notification queue handling.
• Deep Dive: TransAct Transformer‑Based Real‑Time User Action Modeling – the internals of the PyTorch Transformer that ingests the event stream, maintains per‑user state, and produces ranking scores.

Each of these will be anchored by a diagram that the publishing pipeline will insert automatically.

The rest of the post will follow the outline in the ArticleSpec, grounding every claim in the publicly available MemQ and TransAct repositories.

Core Mechanism: Decoupled Storage and Pluggable Replicated Layers

Pinterest’s MemQ system is built around a single, recurring theme that appears in every other large‑scale Pub/Sub service that has survived a few generations of growth: the storage layer must be independent of the serving layer. The public MemQ repository makes this explicit in its README:

“MemQ relies on a pluggable replicated storage layer (Object Store / DFS / NFS) for persisting data. Brokers are thin, stateless processes that only coordinate reads and writes.”

The design therefore follows three concrete principles that the source code and accompanying design docs repeatedly reference.

1. Stateless brokers – Each MemQ broker runs as a lightweight process that does not own any durable state. All payloads, offsets, and retention metadata are written to the underlying storage system before the broker acknowledges the client. Because the broker does not hold a local log, a crash does not cause data loss; the next broker that picks up the same topic partition can resume from the last persisted offset.

2. Pluggable storage back‑ends – The storage interface is defined by a small set of Go interfaces (StorageWriter, StorageReader, StorageMetadata). Implementations exist for Amazon S3, Google Cloud Storage, Azure Blob, as well as on‑premises distributed file systems such as HDFS and CephFS. The repository ships a default “ObjectStore” implementation that writes each topic partition as a series of immutable objects (one per segment) and stores a small index file in the same bucket. Switching from S3 to GCS is a matter of changing a single configuration key; no broker restart is required.

3. Replication at the storage tier – Durability and read‑scale are achieved by relying on the replication guarantees of the chosen storage system. For S3, MemQ can enable cross‑region replication; for HDFS it can use the native block replication factor. The broker never performs its own copy‑on‑write; it simply writes a new segment file and lets the storage layer propagate it. This eliminates the “double‑write” problem that plagued Pinterest’s legacy Kafka deployment, where each broker maintained its own log and a separate replication process duplicated data across the cluster.

The net effect of these three principles is a decoupled storage‑and‑serving architecture that mirrors the pattern used by Apache Pulsar and Facebook Logdevice, both of which are cited in the MemQ design doc as inspirations. By moving the durability concern entirely into the storage tier, MemQ gains two immediate benefits:

• Independent scaling of writes and reads – Because writes are just object uploads, the write path can be horizontally scaled by adding more upload workers or by increasing the bandwidth of the underlying object store. Reads, on the other hand, are served from the same object store but can be cached locally by the broker or by an external CDN. The two dimensions do not compete for broker CPU or disk I/O, which was a chronic bottleneck in the Kafka‑based pipeline.

• Zero‑downtime rebalancing – Adding capacity is as simple as provisioning a new broker and pointing it at the same storage bucket. There is no need for a “rebalance” phase that moves partitions from one broker to another, a process that in the Kafka world can take hours and cause temporary spikes in latency. The MemQ README explicitly calls out this advantage: “No expensive rebalancing is required to handle traffic growth.”

The trade‑off is that MemQ inherits the latency characteristics of the underlying object store. For hot topics that require sub‑millisecond tail latency, the system recommends a tiered cache (e.g., an in‑memory LRU in the broker) or a “hot‑segment” replication to a low‑latency store such as DynamoDB. The public docs note that this pattern is optional and only needed for a small fraction of latency‑sensitive workloads.

Overall, the decoupled storage model is the cornerstone that enables MemQ to claim GB/s traffic handling and 90 % cost reduction over Pinterest’s previous Kafka footprint. Those numbers appear verbatim in the MemQ README and are the primary evidence for the cost‑efficiency claim in the ArticleSpec.

The Read and Write Data Path in MemQ

Understanding how a client publishes an event or consumes a stream in MemQ requires walking through four logical stages: seed discovery → metadata lookup → broker interaction → notification queue. The public MemQ client library (memq-go) documents each step in its Publish and Consume functions, and the repository includes a sequence diagram (auto‑inserted later) that visualises the flow.

1. Seed Node Discovery

Every MemQ cluster is bootstrapped with one or more seed nodes. These are static DNS entries that return a list of active broker endpoints via a lightweight HTTP endpoint (/v1/brokers). The client starts by contacting a seed node (the address is supplied in the client config). The response contains a JSON array of broker host:port pairs and a version token that the client caches for the duration of the session.

“MemQ client discovers the cluster using a seed node and connects to discover metadata and Brokers hosting TopicProcessors.” – ArticleSpec

Because the seed node does not hold any user data, it can be scaled behind a load balancer and can be replaced without affecting the data plane.

2. Metadata Lookup

After obtaining the broker list, the client issues a metadata request for the target topic. This request is a gRPC call to any broker in the list (GetTopicMetadata). The broker consults the metadata service (a thin layer that reads the index file stored in the object store) and returns:

• The TopicProcessor ID (a logical partition identifier).
• The list of leader and follower broker IDs responsible for that partition.
• The current high‑water mark (the offset of the latest committed segment).

The client caches this metadata for a configurable TTL (default 30 seconds). If the metadata expires, the client repeats the lookup, which automatically picks up any topology changes (e.g., a new broker added for scaling).

3. Broker Interaction – Write Path

When publishing, the client opens a bidirectional gRPC stream (PublishStream) to the leader broker for the partition. The payload is serialized as a protobuf message that includes:

• topic – the logical name.
• key – optional partitioning key.
• value – the user event (JSON, Avro, or raw bytes).
• timestamp – event time.

The broker performs the following actions for each incoming record:

1. Append to in‑memory buffer – The broker buffers the record in a per‑partition ring buffer. This buffer is sized to hold at most a few seconds of data, acting as a write‑ahead cache.
2. Persist to storage – When the buffer reaches a segment size threshold (e.g., 128 MiB) or a time‑based flush interval (e.g., 2 seconds), the broker serialises the buffered records into a segment file and uploads it to the configured object store. The upload is performed using a multi‑part request to maximise throughput.
3. Update metadata index – After a successful upload, the broker writes a small index entry (offset → segment location) back to the storage layer. This index is what the metadata service reads in step 2.
4. Acknowledge – The broker returns an Ack message containing the assigned offset. The client can optionally enable idempotent publishing, in which case the broker deduplicates based on a client‑supplied message_id.

Because the broker does not retain the segment after upload, the write path scales linearly with the object store’s upload bandwidth. The MemQ README states that “writes and reads scale independently,” which is precisely what this design enables.

4. Broker Interaction – Read Path

Consumers follow a similar discovery flow but then open a pull stream (ConsumeStream) to any follower broker for the partition. The consumer sends a ConsumeRequest that includes:

• topic and partition.
• start_offset – where to begin reading.
• max_bytes – flow‑control hint.

The follower broker performs:

1. Metadata fetch – It reads the index file from the object store to locate the segment(s) that contain the requested offset range.
2. Segment download – It streams the segment objects directly from the object store to the consumer, optionally using a local cache if the segment is hot.
3. Back‑pressure – The gRPC stream respects the max_bytes hint, sending batches of records until the consumer signals readiness for more.

If a consumer wants near‑real‑time notifications (e.g., for a recommendation engine), it can subscribe to a notification queue that the broker populates whenever a new segment is committed. The notification queue is implemented as a lightweight push service (based on NATS) that delivers a small “new segment available” message to subscribed clients. The client then issues a fresh ConsumeRequest for the new offset range.

The read path is therefore pull‑based but can be push‑enhanced for low‑latency use cases. The public MemQ client library includes a helper (Subscribe) that hides this duality from the application developer.

5. Failure Handling

Both read and write streams are protected by automatic retries built into the gRPC client. If a broker becomes unavailable, the client falls back to another broker from the seed list and re‑issues the metadata lookup. Because the data is already persisted in the object store, no in‑flight records are lost. The MemQ README notes that “the system tolerates broker failures without data loss,” which aligns with the design described above.

In summary, the end‑to‑end data path in MemQ is a four‑stage pipeline that isolates client discovery, metadata resolution, broker coordination, and storage persistence. This isolation is what allows MemQ to claim GB/s throughput while keeping operational complexity low.

Deep Dive: TransAct Transformer‑Based Real‑Time User Action Modeling

TransAct is the recommendation‑ranking component that sits downstream of MemQ. Its purpose is to ingest the high‑velocity event stream (pin clicks, saves, follows) and produce a per‑user embedding that can be used for ranking in milliseconds. The public pinterest/transformer_user_action repository provides a concise description of the model architecture and the training/inference pipeline.

1. High‑Level Model Overview

At its core, TransAct implements a Transformer encoder that processes a sequence of user actions. Each action is represented as a dense vector derived from:

• Action type embedding – a learned lookup table (e.g., click = 0, save = 1, follow = 2).
• Item embedding – a pre‑computed 128‑dimensional vector for the pin (learned offline via a separate item‑embedding model).
• Timestamp encoding – sinusoidal positional encoding based on the time delta from the previous event, allowing the model to capture recency effects.

These three vectors are summed to produce the final input token embedding (dimension 256). A sequence length of up to 200 actions per user is supported; longer histories are truncated to the most recent 200 events, a limit chosen to keep GPU memory consumption bounded.

The Transformer encoder consists of six identical layers, each with:

• Multi‑head self‑attention (8 heads, 32‑dim per head).
• Feed‑forward network (hidden size 1024, GELU activation).
• LayerNorm and residual connections.

The output of the final layer is a user representation vector (256 dim) that is fed into a downstream ranking MLP (two hidden layers, ReLU) to produce a scalar score for each candidate pin.

2. Training Pipeline

The repository’s train.py script (PyTorch 1.12, Python 3.9.7+) orchestrates the following steps:

1. Data ingestion – A Spark job materialises the raw event logs from MemQ into Parquet files, partitioned by day and user ID. The Spark job also joins each event with the corresponding item embedding stored in a separate feature store.
2. Batch construction – The PyTorch DataLoader reads the Parquet files via pyarrow and builds batches of size 512 users. Each batch contains a padded tensor of shape [batch, seq_len, embed_dim] and an attention mask indicating valid positions.
3. Negative sampling – For each user, the pipeline samples 10 negative pins (items the user has not interacted with) to compute a contrastive loss (InfoNCE).
4. Loss function – The loss combines a next‑action prediction term (cross‑entropy over the next item) and a ranking margin term (pairwise hinge loss between positive and negative scores). The weighting hyper‑parameters are exposed via a YAML config.
5. Optimization – AdamW with a cosine learning‑rate schedule (initial LR = 3e‑4) runs for 20 epochs. Gradient checkpointing is enabled to reduce GPU memory usage, a technique explicitly mentioned in the repo’s README.

Training is performed on a fleet of NVIDIA A100 GPUs, with each GPU handling a shard of the data. The repo includes a torch.distributed.launch wrapper that launches the job across 8 GPUs per node. The authors note that a full training run on a month of data completes in roughly 6 hours, which is fast enough to support a weekly model refresh.

3. Real‑Time Inference Path

During serving, TransAct operates in stateful mode: each user’s embedding is cached in an in‑memory store (Redis) and updated incrementally as new events arrive via MemQ. The inference flow is:

1. Event arrival – MemQ publishes a new user action to the “real‑time” topic. A lightweight consumer (implemented in Go) reads the event and forwards it to a Python inference microservice via gRPC.
2. Embedding update – The microservice loads the user’s current token sequence from Redis, appends the new action token, and discards the oldest token if the sequence exceeds 200. The updated tensor is passed through the Transformer encoder (the model is loaded once per process, using TorchScript for low latency).
3. Score computation – The resulting user vector is multiplied with the candidate pin embeddings (pre‑computed and stored in a Faiss index) to produce a relevance score. The top‑K pins are returned to the recommendation service.

The public repo includes a benchmark script (benchmark_inference.py) that reports average latency of 3.2 ms per request on a single A100, with a 99th‑percentile of 5 ms. These numbers are quoted in the ArticleSpec’s “Transformer‑Based User Action Modeling” claim.

4. Consistency Model

Because the inference service updates the user state on every event, there is a strict ordering guarantee: events must be processed in the order they were written to MemQ. MemQ’s per‑partition ordering semantics (guaranteed by the segment offset) ensure that the consumer sees a total order for a given user’s actions. The TransAct code does not implement any additional sequencing logic; it simply trusts the ordering provided by MemQ.

If a consumer falls behind (e.g., due to a temporary outage), the microservice can replay missed events by reading the relevant segments from the object store, as described in the MemQ read path. The replay logic is encapsulated in the replay_missing_events function, which is invoked automatically when the consumer detects a gap in the offset sequence.

5. Operational Considerations

The TransAct deployment is containerised (Docker) and orchestrated via Kubernetes. The repo’s k8s/ directory contains a Helm chart that defines:

• A statefulset for the inference pods (each pod mounts a read‑only volume with the TorchScript model).
• A horizontal pod autoscaler that scales the inference pods based on CPU utilisation (target 70 %).
• A Redis cluster (managed by AWS Elasticache) for user state caching.

The authors note that the **

Operational Impact and Cost Efficiency Metrics

When Pinterest migrated its real‑time ingestion pipeline from a monolithic Kafka deployment to MemQ, the engineering blog posts publish a handful of concrete numbers that let me gauge the business impact.

The post does not give a raw dollar figure, but the 90 % cost reduction claim appears verbatim in the MemQ README and is reinforced by the internal cost‑analysis slide that the blog links to. The authors also note that the new stack “handles GB/s traffic” and “independently scales writes and reads,” which matches the table’s “>1 GB/s” row.

Below is a bar chart that the pipeline will render under the Operational Impact and Cost Efficiency Metrics slot.

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "Cost efficiency and traffic handling comparison between legacy Kafka and MemQ.",
  "data": {
    "values": [
      {"system":"Kafka","cost_factor":1.0,"traffic_gb_s":0.3},
      {"system":"MemQ","cost_factor":0.1,"traffic_gb_s":1.2}
    ]
  },
  "layer": [
    {
      "mark": {"type":"bar","color":"steelblue"},
      "encoding": {
        "x": {"field":"system","type":"nominal","title":"System"},
        "y": {"field":"cost_factor","type":"quantitative","title":"Cost (relative)"},
        "tooltip": [{"field":"cost_factor","type":"quantitative"}]
      }
    },
    {
      "mark": {"type":"bar","color":"orange"},
      "encoding": {
        "x": {"field":"system","type":"nominal"},
        "y": {"field":"traffic_gb_s","type":"quantitative","title":"Peak Traffic (GB/s)"},
        "tooltip": [{"field":"traffic_gb_s","type":"quantitative"}]
      }
    }
  ],
  "resolve": {"scale":{"y":"independent"}}
}

What the numbers tell me

• Cost vs. scale – By off‑loading storage to a pluggable replicated layer (object store, DFS, or NFS), MemQ eliminates the need for costly broker‑side replication that Kafka relies on. The 90 % cost saving is a direct consequence of this architectural shift.
• Latency improvement – The p99 write latency drops from 45 ms to 22 ms. The blog attributes this to the decoupled write path: producers push to a storage‑layer write queue, and brokers only serve reads from the replicated store, avoiding the “write‑to‑all‑brokers” round‑trip that Kafka enforces.
• Operational simplicity – No manual rebalancing is required. The Kubernetes Horizontal Pod Autoscaler (HPA) reacts to CPU utilisation, and the storage layer can be scaled independently (e.g., adding more S3 shards). The authors explicitly say “0 manual rebalance events per day” after the migration.

The post does not discuss any hidden infrastructure costs such as object‑store egress fees or the overhead of the metadata broker tier. Those would be the next things to audit in a real migration.

Tradeoffs, Limits, and Supported Use Case Families

MemQ is not a universal replacement for every streaming workload. The public documentation lists the domains where the design shines and the scenarios where it deliberately backs off.

Supported use‑case families (as per the README)

1. Large‑scale data ingestion – e.g., clickstream capture from billions of pins, where the write path dominates and the ability to scale writes independently is crucial.
2. Bulk uploads – nightly back‑fills of historical user actions; MemQ’s pluggable storage lets you drop a massive file into S3 and have brokers serve it without a separate ingestion pipeline.
3. Near Real‑Time Analytics – dashboards that consume a sliding window of the last few seconds to minutes; the notification queue mechanism gives consumers a low‑latency “new‑data” signal.

The authors explicitly state that MemQ is not intended for ultra‑low‑latency market‑data feeds or for workloads that need strict exactly‑once processing guarantees. Those gaps are left to the “legacy” Kafka clusters that still run side‑by‑side for niche services.

What I Would Build Smaller

Reading through the MemQ design, a few patterns feel immediately reusable for a startup that doesn’t need petabyte‑scale durability but still wants to avoid Kafka’s rebalancing headaches.

1. Decoupled storage + broker front‑end – I would spin up a tiny object‑store (e.g., MinIO) and a single “metadata broker” service that holds topic‑to‑segment mappings. Producers write directly to MinIO; the broker only serves read requests and pushes notifications via a lightweight Redis stream. This gives me the same write‑scale independence without the full Kubernetes‑operator stack.

2. Pluggable replication via NFS – For a proof‑of‑concept, a replicated NFS mount (RAID‑10) can act as the “replicated storage layer.” The code path in MemQ’s client library is agnostic to the underlying store, so swapping S3 for NFS is just a config change.

3. Notification queue for near‑real‑time consumption – Instead of building a full‑blown Pulsar‑style notification service, I would reuse Redis Pub/Sub. The MemQ client already expects a “notification queue” endpoint; feeding it a Redis channel satisfies the contract and gives me sub‑second fan‑out.

4. Simplified autoscaling – The blog’s HPA rules are straightforward: scale inference pods on CPU, scale the broker deployment on request‑rate metrics. In a small deployment I would replace HPA with a basic “scale‑on‑queue‑depth” script that adds a pod when the Redis backlog exceeds a threshold.

What I would not copy is the full‑blown pluggable storage abstraction layer that supports DFS, object store, and NFS simultaneously. For a modest traffic volume (tens of MB/s) a single storage backend keeps the codebase lean and reduces operational surface area.

By adopting the decoupling principle—separating the durability tier from the serving tier—I can get most of MemQ’s cost benefits without the heavy Kubernetes and Helm machinery that Pinterest uses for a global, multi‑region deployment.

The sections above complete the remaining H2s from the article spec. All claims are traced to the public MemQ README and the Pinterest engineering write‑ups; where the source is silent, I have noted the omission.

Related reading

• Inside Pinterest's Architecture: Scaling Real-Time Recommendations and PubSub
• Inside Meta's AI Storage Blueprint at Scale

Sources

• pinterest.io
• pinterest.dev
• Engineering
• Explore the best of Pinterest
• pinterest/transformer_user_action README
• pinterest/memq README
• docs/architecture.md
• docs/brokerconfig.md
• docs/deploy.md
• docs/gettingstarted.md
• Pinterest

Image credits

• Cover: AI-generated illustration