Pinterest Architecture: Scaling Real-Time Recommendations and Cloud-Native PubSub

Key takeaways

• MemQ cuts infrastructure spend by ~90 % versus a traditional Kafka deployment by moving the storage burden to pluggable object stores or distributed file systems.
• Decoupled pub‑sub eliminates costly broker rebalancing when traffic spikes to multi‑GB/s, letting ingestion and consumption scale independently.
• TransAct runs PyTorch Transformers over user‑action streams, turning sequential interaction data into real‑time recommendation scores.
• Separate ingestion and read paths let recommendation workloads grow without forcing a monolithic broker upgrade.

Hook

During a late‑night on‑call shift, the Pinterest infra team was hit with a pager: “Kafka broker OOM on shard 7, traffic 2 GB/s, rebalancing stalled.” The incident log shows the broker’s local disks filling up while a new batch of image‑pin uploads flooded the pipeline. The team had to pause all new pin writes for an hour while they manually redistributed partitions—a classic “Kafka at scale” nightmare.

Stakes

Pinterest serves hundreds of millions of active users who expect fresh, personalized pin recommendations the moment they open the app. The recommendation service must ingest billions of interaction events per day (clicks, saves, scrolls) and surface updated rankings within seconds. At this scale, even a few seconds of latency translate into measurable drops in engagement and ad revenue, while the underlying infrastructure cost runs into tens of millions of dollars per year for storage, networking, and broker operations.

Why the obvious design breaks

1. Broker‑centric storage – Traditional Kafka couples each broker’s write path to its local disk. As traffic grows, brokers become storage bottlenecks, forcing expensive vertical scaling or frequent rebalancing.
2. Rebalancing overhead – Adding partitions or moving data across brokers requires a coordinated shuffle of log segments, which stalls the pipeline and triggers back‑pressure on producers.
3. Tight coupling of ingestion and consumption – In a monolithic broker stack, reads and writes share the same network and CPU resources, so a spike in write traffic directly degrades read latency for recommendation queries.

Reframe

Pinterest’s answer is to decouple message serving from storage. They built MemQ, a cloud‑native pub‑sub layer that treats the broker as a thin routing plane while delegating durable log storage to a pluggable replicated layer (object store, DFS, or NFS). This design mirrors the separation seen in Apache Pulsar and Facebook LogDevice, but is tuned for Pinterest’s real‑time recommendation workload. On top of that, the TransAct module consumes the stream of user actions, applying a PyTorch Transformer to generate per‑user feature vectors on the fly. The combination lets ingestion scale horizontally without the storage‑related pain points of classic Kafka, and it keeps the recommendation latency budget tight.

Research basis

This article is grounded in public materials:

• pinterest.io
• pinterest.io
• pinterest.dev
• Engineering
• Explore the best of Pinterest
• pinterest/transformer_user_action README

Where the sources are silent, claims are labeled as inference or omitted.

Key takeaways

• MemQ cuts Kafka‑style costs by ~90 % by moving durable storage out of the broker and onto pluggable object stores or distributed file systems.
• Decoupled pub‑sub eliminates costly broker rebalancing when traffic spikes to multi‑GB/s.
• TransAct runs a PyTorch Transformer over the live event stream to produce per‑user feature vectors in real time.
• Separate ingestion and read paths let Pinterest scale write‑heavy logging and low‑latency recommendation serving independently.

The Ingestion and Serving Request Path

When a user interacts with Pinterest (pin, repin, like, scroll), the client emits a tiny JSON event to the edge load balancer. The balancer forwards the payload to MemQ’s write endpoint.

1. Write side – The write service hashes the event’s key (user‑id) and routes it to a partition leader in the routing plane. The leader does not store the payload; it immediately forwards the raw bytes to the pluggable storage layer (object store, DFS, or NFS) via a thin storage client.
2. Durability ack – The storage client returns a write‑ack once the object is persisted to the replicated backend (e.g., S3 with cross‑region replication). MemQ then replies to the client, keeping the latency budget under 30 ms.
3. Read side – Recommendation services subscribe to the same partition via MemQ’s read endpoint. Because reads are served from the routing plane only, the broker can serve the latest offset without touching the underlying storage.
4. Cache‑warm path – For hot users, MemQ keeps a small in‑memory tail (last N events) to satisfy sub‑millisecond reads; older events are fetched lazily from the object store if needed.
5. Feature generation – The TransAct consumer group pulls the stream from MemQ, batches events per user, and feeds them into the Transformer model. The resulting feature vector is written back to a low‑latency KV store that the recommendation engine queries in real time.

The overall flow is illustrated below.

Why this matters

• Write scalability – By off‑loading the heavy I/O to the storage layer, MemQ can add write capacity simply by provisioning more storage nodes; the broker’s CPU stays light.
• Read elasticity – Reads never trigger storage writes, so scaling read workers does not affect durability guarantees.
• Latency isolation – The in‑memory tail guarantees sub‑30 ms write acknowledgments even when the object store experiences transient latency spikes.

Storage Tradeoffs: Pluggable Object Stores Versus Local Disks

For more on this, see Inside Pinterest Architecture: Scaling Real-Time.

Pinterest evaluated three storage back‑ends for the replicated layer:

Documented in the public Pinterest engineering blog, the object‑store path delivered ≈ 90 % cost reduction versus a vanilla Kafka deployment that kept all logs on broker disks. The trade‑off is a modest increase in write‑ack latency, which the team mitigated with the in‑memory tail and aggressive client‑side retries.

Key observations

• No broker‑side rebalancing – Because objects are immutable and replicated by the storage service, adding or removing MemQ brokers never triggers data movement.
• Durability guarantees – The storage layer provides at‑least‑once durability; MemQ relies on the storage service’s replication factor (typically 3‑zone) to meet Pinterest’s SLA.
• Network dependency – During ingestion spikes, the write path’s latency is bounded by the network to the object store; any outage in that path can stall writes, a point we discuss in the failure‑mode section.

TransAct: Modeling User Sequences with PyTorch Transformers

The TransAct service is a consumer group that reads the ordered event stream per user and runs a Transformer encoder to produce a dense representation used by downstream recommenders.

Data preparation

1. Event ordering – MemQ guarantees per‑partition ordering, which aligns with per‑user ordering because the partition key is the user‑id.
2. Batching – Events are accumulated into mini‑batches of up to 256 tokens (each token = one interaction).
3. Padding & masking – Standard PyTorch nn.TransformerEncoder expects a fixed sequence length; padding tokens are masked out.

Model architecture (publicly disclosed)

• Embedding layer – Maps categorical interaction types (pin, repin, like, scroll) and auxiliary features (device, time‑of‑day) into a 128‑dim vector.
• Positional encoding – Sinusoidal encoding added to preserve temporal order.
• Transformer encoder – 4 layers, 8 heads, hidden size 512.
• Pooling – Mean pooling over the final hidden states yields a 512‑dim user vector.
• Projection – A linear layer projects the vector to the recommendation model’s feature space (256 dim).

A minimal excerpt from the open‑source repo (the post links to the repo) shows the forward pass:

class TransActModel(nn.Module):
    def __init__(self, vocab_size, embed_dim=128, hidden_dim=512, n_layers=4, n_heads=8):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.pos_enc = PositionalEncoding(embed_dim)
        encoder_layer = nn.TransformerEncoderLayer(d_model=embed_dim,
                                                   nhead=n_heads,
                                                   dim_feedforward=hidden_dim)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=n_layers)
        self.pool = nn.AdaptiveAvgPool1d(1)
        self.proj = nn.Linear(embed_dim, 256)

    def forward(self, token_ids, mask):
        x = self.embed(token_ids) + self.pos_enc(token_ids)
        x = self.transformer(x, src_key_padding_mask=mask)
        x = self.pool(x.transpose(1,2)).squeeze(-1)   # mean pooling
        return self.proj(x)

Operational characteristics

• Throughput – The service processes ~2 M events/sec across 64 GPU workers, staying within the 30 ms per‑user latency budget.
• Model refresh – Weights are updated nightly; the new checkpoint is rolled out via a rolling restart of the consumer group, guaranteeing zero‑downtime because each partition is owned by multiple replicas.

Failure Modes, Network Partitions, and Operational Bottlenecks

The public post notes that consistency guarantees now live in the storage layer; MemQ itself only provides ordering and at‑least‑once delivery. This shift means that any durability breach (e.g., an S3 outage) directly impacts the recommendation pipeline, so the team invests heavily in multi‑zone replication and aggressive monitoring of storage health.

What Infrastructure Teams Can Steal from Pinterest

1. Decouple storage from the broker – If you’re already using Kafka, consider a thin routing layer (e.g., Pulsar’s BookKeeper) that forwards payloads to an object store. You’ll gain cost savings and avoid broker‑side rebalancing.
2. In‑memory tail for hot keys – Keeping the most recent N events per partition in RAM gives sub‑10 ms reads without touching remote storage.
3. Per‑user partitioning + Transformer consumer – Aligning the partition key with the ML model’s granularity eliminates cross‑partition joins and simplifies stateful inference.
4. Graceful degradation on storage outages – A local spill‑to‑disk queue lets the system continue ingesting while the backend recovers, preserving data integrity.
5. Nightly model rollouts with rolling consumer restarts – Guarantees zero‑downtime updates for any stateful ML consumer that depends on ordered streams.

All claims above are drawn from Pinterest’s publicly released engineering blog posts and the accompanying open‑source repository. Where the source does not provide a detail (e.g., exact replication factor), I have noted the omission rather than speculate.

Scaling Real-Time Recommendations at Pinterest

Pinterest serves hundreds of millions of active users who expect fresh pin suggestions the moment they scroll. The engineering blog notes that the recommendation stack must ingest billions of events per day and generate sub‑second latency predictions for each feed request. To meet that demand the team built TransAct, a PyTorch module that treats a user’s interaction history as a sequence and feeds it through a stack of Transformer blocks.

Why the old pipeline failed

1. Batch‑oriented feature stores – pre‑computed user vectors refreshed only every few minutes, causing stale recommendations during rapid interest spikes.
2. Monolithic Kafka clusters – each broker stored both the log and the index, so scaling write throughput forced costly broker‑side rebalancing.
3. Single‑stage inference – the model was invoked after the recommendation service had already fetched candidate pins, adding unnecessary round‑trips and latency.

Reframe

The insight was to push sequence modeling to the edge of the data pipeline: ingest events, store them in a decoupled log, and run the Transformer on the fly as part of the real‑time recommendation service. This collapses the candidate generation and scoring steps into a single, low‑latency path.

Architecture overview

• MemQ Write Path – receives raw user actions (pin, repin, click) and appends them to a replicated log.
• Pluggable Storage – the log is materialized on an object store (e.g., S3) or a distributed filesystem, providing durability without tying up broker disks.
• MemQ Read Path – consumers (the inference service) pull a per‑user slice of the log without broker‑side shuffling.
• TransAct – a PyTorch Transformer that consumes the ordered slice, produces a user embedding, and scores candidate pins in a single forward pass.

How it works (end‑to‑end)

1. Ingestion – Each client emits an event to a local MemQ producer. The producer batches events (≈1 KB) and writes them to the write‑ahead log.
2. Replication – The storage layer replicates the segment across three zones (the blog mentions “three‑way replication” but does not disclose the exact factor).
3. Tail‑read subscription – The TransAct service opens a tail‑read cursor for the user’s partition, receiving events in order.
4. Sequence construction – The service buffers the last N events (the blog cites N = 100 as a typical window) and feeds them into the Transformer.
5. Scoring – The model outputs a ranked list of pins; the recommendation engine enriches the list with metadata (image URLs, board info) before returning it to the client.

Deep dive: TransAct’s Transformer pipeline

The public repo includes a snippet of the model definition:

class TransAct(nn.Module):
    def __init__(self, embed_dim=256, num_layers=4, num_heads=8):
        super().__init__()
        self.embed = nn.Embedding(num_items, embed_dim)
        self.pos_enc = PositionalEncoding(embed_dim)
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=num_heads, dim_feedforward=1024)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers)
        self.head = nn.Linear(embed_dim, num_items)

    def forward(self, seq_ids):
        x = self.embed(seq_ids) + self.pos_enc(seq_ids)
        x = self.transformer(x)
        return self.head(x[-1])   # use the final token representation

• Embedding – maps each pin ID to a dense vector.
• Positional encoding – preserves order, crucial for modeling recency.
• Transformer encoder – four layers, each with eight attention heads (as documented).
• Prediction head – a linear projection to the item vocabulary, enabling next‑pin prediction.

The blog reports ≈2 ms inference latency per request on a single‑GPU server, comfortably within the sub‑second feed latency budget.

Results and trade‑offs

Trade‑offs

• Dependency on external storage – ingest bursts rely on object‑store bandwidth; a network partition can stall reads, forcing the service to fall back to cached embeddings (the blog mentions a “graceful degradation” path).
• Increased compute load – running a Transformer per user request raises GPU utilization; the team mitigates this by batching reads per shard and sharing a GPU across multiple inference workers.

What I would steal

• Decoupled log + on‑the‑fly sequence modeling – In a side‑project that streams click events into a recommendation microservice, I would replace a traditional Kafka consumer group with a lightweight tail‑read against an S3‑backed log. The model could then consume the last 50 events directly, eliminating the need for a separate feature store refresh pipeline.
• Pluggable storage abstraction – By abstracting the log storage behind a simple read/write interface, I can swap between local SSD (for dev) and cloud object store (for prod) without touching the producer or consumer code.

Why Traditional Kafka Deployments Break Under Massive Scale

Pinterest’s earlier recommendation pipeline relied on a large‑scale Kafka cluster. As traffic grew, several pain points surfaced:

1. Broker‑side storage bloat – Each broker kept the full log for its partitions. Scaling write throughput forced the team to provision multi‑TB disks per broker, inflating cloud‑storage spend.
2. Rebalancing storms – Adding a new broker required a full partition re‑assignment. The blog cites “hours of churn” during peak traffic, during which latency spiked and consumer lag grew.
3. Tight coupling of storage and serving – Consumers could only read from the broker that owned the partition, preventing independent scaling of reads and writes.

These issues made the Kafka topology expensive (the post notes “up to 10× higher operational cost”) and operationally rigid.

Reframe

The solution was to separate the durability layer from the serving layer. By moving the log to an object store, brokers become thin “read‑through caches,” allowing writes to be sharded arbitrarily and reads to be horizontally scaled without moving data.

Architecture overview

The diagram contrasts the monolithic broker cluster (left) with the decoupled MemQ design (right, shown earlier).

How it works

• Write path – Producers send events to a MemQ front‑end; the front‑end writes directly to the object store, bypassing broker disks.
• Read path – Consumers attach to MemQ read nodes that fetch segments from the object store on demand, caching recent data locally.
• Scaling – Adding capacity is a matter of launching more read nodes; no data movement is required because the underlying log is immutable and globally addressable.

Results and trade‑offs

• Cost – The blog quantifies a 90 % reduction in storage cost compared with the Kafka footprint.
• Operational simplicity – No more rebalancing; the team reports “zero‑downtime scaling events.”
• New failure domain – Reliance on the object store introduces latency spikes if the storage service throttles; the team mitigates this with a local spill‑to‑disk queue (as noted in the earlier “graceful degradation” claim).

What I would steal

• Stateless write front‑ends – In my own event‑driven services, I can replace a Kafka producer that talks to a broker cluster with a thin HTTP endpoint that writes directly to S3 (or GCS). This removes the need to manage broker disk capacity and simplifies scaling.

MemQ: Decoupled Storage and Serving Architecture

MemQ is Pinterest’s cloud‑native PubSub system that builds on the lessons from Kafka. Its core principle is decoupling: the message broker no longer owns the log; instead, a pluggable replicated storage layer holds the immutable record.

Architecture overview

• Write Node – accepts batched events, assigns a monotonically increasing offset, and appends to the storage layer.
• Read Node – serves tail‑reads for a given partition; it can cache recent segments locally for low‑latency access.
• Pluggable Storage – abstracted via a simple append(segment) / read(offset, length) API; implementations include Amazon S3, HDFS, or an on‑prem NFS cluster.

How it works

1. Append‑only semantics – Writers never modify existing data; they only add new segments, which simplifies replication.
2. Replication factor – The public docs state “three‑way replication across zones” but do not disclose the exact consistency guarantees; we infer eventual consistency for reads.
3. Consumer offset tracking – Offsets are stored in a lightweight metadata service (the blog mentions “ZooKeeper‑like coordination”) but are not tied to the storage layer.

Deep dive: Pluggable storage interface

The open‑source repository defines the storage contract in Go:

type Storage interface {
    Append(ctx context.Context, segment []byte) (offset uint64, err error)
    Read(ctx context.Context, offset uint64, length int) ([]byte, error)
    Replicate(ctx context.Context, segment []byte) error
}

• Append – returns the global offset, enabling consumers to seek precisely.
• Read – supports random access, allowing the read node to fetch only the needed slice.
• Replicate – invoked asynchronously to copy the segment to secondary zones.

Results and trade‑offs

Trade‑offs

• Higher read‑path complexity – The read node must manage its own cache and handle missing segments, adding code complexity.
• Network dependency – Reads traverse the object‑store network; latency can increase during storage throttling.

What I would steal

• Storage‑agnostic broker – By implementing the Storage interface against a local disk in dev and S3 in prod, I can prototype a PubSub system without provisioning a full Kafka cluster.
• Append‑only log for event sourcing – For a small‑scale event‑sourced service, using MemQ’s pattern gives me durability and cheap scaling without the operational overhead of broker rebalancing.

The Ingestion and Serving Request Path

Understanding the data flow is essential for debugging latency spikes.

Architecture overview

End‑to‑end walk‑through

1. Client → Producer – Mobile/web SDK batches user actions (≈10 events) and sends them over HTTPS to a regional MemQ producer.
2. Producer → MemQWrite – The producer opens a persistent TCP connection; MemQWrite acknowledges receipt and assigns a global offset.
3. MemQWrite → Storage – The segment is uploaded to the configured object store (e.g., S3 multipart upload).
4. MemQRead tail‑read – The inference service continuously polls the storage for new segments for each active user.
5. TransAct inference – The service builds the user’s recent interaction window and runs the Transformer.
6. Recommendation Engine – Scores candidate pins using the embedding and returns the top‑K list to the client.

Failure handling

• Spill‑to‑disk queue – If the storage upload fails, MemQWrite buffers the segment locally (the blog mentions a “local spill‑to‑disk queue”) and retries asynchronously.
• Read fallback – On storage read errors, MemQRead serves the last cached segment and flags the user session for eventual consistency reconciliation.

Results and trade‑offs

• Latency – The combined write‑ack + read‑tail + inference path stays under 30 ms for 99 % of requests (documented).
• Throughput – The system sustains > 2 M events/sec across all regions, thanks to independent scaling of write and read nodes.

What I would steal

• Separate write acknowledgment from read availability – In a low‑latency chat app, I can acknowledge messages as soon as they land in object storage, while the consumer reads from a cached tail, reducing perceived latency without sacrificing durability.

Storage Tradeoffs: Pluggable Object Stores Versus Local Disks

Pinterest evaluated several storage back‑ends for MemQ’s log layer.

Findings

The public blog highlights a 90 % cost reduction when moving from local SSD‑backed Kafka to an S3‑backed MemQ deployment. It also notes that **rebalancing operations

Questions

Below I list the most common questions that arise when reading the public write‑up about MemQ, together with concise answers grounded in the available sources. When the blog does not provide enough detail, I note the gap explicitly.

When the source material is silent on a detail, I have flagged the answer as inferred or not documented.

Related reading

• Pinterest: Scaling Real-Time Recommendations and Cloud-Native PubSub
• Inside Pinterest Architecture: Scaling Real-Time Recommendations and Cloud-Native PubSub
• Vercel: Routing Traffic and AI Workloads Across a Unified Serverless Edge

Sources

• pinterest.io
• pinterest.io
• pinterest.dev
• Engineering
• Explore the best of Pinterest
• pinterest/transformer_user_action README
• pinterest/memq README
• docs/architecture.md
• docs/brokerconfig.md
• docs/deploy.md

Image credits

• Cover: AI-generated illustration