Inside LinkedIn's Engineering Architecture: From Rest.li to Venice and gRPC

Navigating the 404 and Legacy Discovery Blocks

When I typed https://linkedin.dev/docs into my browser expecting a quick reference for LinkedIn’s internal APIs, I was met with a stark 404: Not Found page. The response was just the generic “The request was not found” message, a dead‑end for anyone trying to discover the current state of LinkedIn’s service‑to‑service contract layer. The same URL that once pointed to a living catalog of Rest.li endpoints now returns nothing, a silent indicator that the platform that powered LinkedIn’s RPC layer for years is being retired.

The public repository for Rest.li makes the same point explicit: the project is deprecated and will be archived on July 22, 2026. The README notes that LinkedIn has already begun an internal migration to gRPC, and that “gRPC is not a drop‑in replacement for Rest.li.” The combination of a vanished documentation portal and an official deprecation notice tells a clear story—LinkedIn’s legacy discovery mechanism is on its way out, and the engineering effort to replace it is already underway.

---

Why the Naive or Legacy Approach Fails

1. Static discovery via Rest.li docs is gone – Without a living documentation endpoint, developers cannot rely on auto‑generated client stubs or runtime service discovery.
2. Rest.li’s RPC model is monolithic – Each service publishes a fixed set of resources; extending or versioning them requires coordinated schema changes across the fleet.
3. Performance ceiling – Rest.li’s HTTP‑based transport adds latency compared with the binary, multiplexed streams that gRPC offers.
4. Language support – Rest.li’s code generation is tightly coupled to Java (Pegasus); teams using Python, Go, or Rust must fall back to hand‑written clients.
5. Streaming limitations – Long‑running data feeds (e.g., real‑time feed updates) are awkward to implement on top of Rest.li’s request/response pattern.

The LinkedIn README for Rest.li explicitly lists the migration to gRPC as a move toward “better performance, support for more programming languages, streaming, and a robust open source community.” The legacy approach therefore breaks on three fronts: discoverability, scalability, and ecosystem fit.

---

High-Level Architecture or Infrastructure Stack

LinkedIn’s data platform is organized around two broad categories:

• Source‑of‑Truth (SoT) systems – Primary stores that own the canonical user‑generated writes (e.g., profile service, connection graph, activity logs).
• Derived data stores / indexes – Secondary systems that materialize read‑optimized views for low‑latency queries, analytics, or recommendation pipelines.

The migration from Rest.li to gRPC sits at the service‑to‑service RPC layer that connects the SoT services to each other and to the derived stores. The stack, as described in the public READMEs, consists of:

Layer	Component	Role
0 – Transport	gRPC (replacing Rest.li)	Binary, multiplexed RPC with streaming support; language‑agnostic client generation.
1 – Change Capture	Databus	Captures mutations from SoT systems and publishes them to downstream pipelines.
2 – Derived Store Platform	Venice	Planet‑scale key‑value store that materializes derived data (e.g., follower counts, feed relevance scores).
3 – Caching / Front‑End	Various CDN / edge caches (not named in the pack)	Serve read‑heavy traffic; invalidate on mutation events from Databus.

The relationship is a classic source‑of‑truth vs. derived store synchronization pattern. Primary writes flow into the SoT, Databus picks up the change events, and Venice ingests those events to keep its derived tables fresh. Client applications—whether they are internal services or external front‑ends—read from Venice or the cache layer, never hitting the SoT for high‑throughput read paths.

Because the public pack does not detail every intermediate component (e.g., specific stream processors or orchestration tools), I will limit the description to the named pieces: gRPC, Databus, and Venice. The post does not say how LinkedIn orchestrates the migration steps beyond “advanced automation,” so I will not speculate on the exact tooling.

---

(The sections that follow—Core Mechanism, Request/Data/Control Path, Deep Dive on Databus and Venice, Results, Tradeoffs, and What I Would Build Smaller—will each be accompanied by a diagram inserted automatically in the slots defined in the research pack.)

Core Mechanism or Reframe

The fundamental insight behind LinkedIn’s migration is a clean separation between primary transactional writes and derived, read‑optimized stores. In the legacy stack, Rest.li services often performed both the write to the source‑of‑truth (SoT) and the materialization of secondary indexes in the same request path. That coupling forced every write to wait for downstream pipelines, inflating tail latency and making it hard to evolve the read side independently.

The new design reframes the problem as a two‑phase data flow:

1. Phase 1 – Write to the SoT – A gRPC service receives the client mutation and persists it to the authoritative store (e.g., a relational DB or a key‑value store). The service returns success as soon as the write is durably logged.
2. Phase 2 – Asynchronous Derivation – A change‑data‑capture (CDC) layer (Databus) streams the mutation events to downstream processors that materialize the data into secondary stores such as Venice. Those stores power high‑throughput read paths, cache layers, and analytics pipelines.

By decoupling the two phases, LinkedIn can:

• Scale reads independently – Venice is tuned for low‑latency, planet‑scale lookups, while the SoT can focus on strong consistency and durability.
• Upgrade the RPC layer without touching downstream stores – gRPC replaces Rest.li at the service boundary; the downstream derivation pipeline remains unchanged.
• Introduce richer transformation logic – Custom pipelines can enrich, filter, or aggregate data before it lands in Venice, without impacting the write latency budget.

The post does not describe any “dual‑write” fallback or a hybrid mode where Rest.li and gRPC coexist on the same endpoint. All references point to a wholesale migration driven by “advanced automation,” implying that the separation is enforced at the API contract level rather than through runtime feature flags.

---

Request/Data/Control Path

Below is the end‑to‑end flow for a typical member profile update:

1. Client → gRPC Service

   • The mobile or web client issues a UpdateProfile RPC over HTTP/2.
   • The gRPC stub performs client‑side load balancing and TLS termination, then forwards the request to the appropriate service instance.

2. Service → Source‑of‑Truth (SoT)

   • The service validates the payload, writes the mutation to the primary store (e.g., a MySQL cluster or a distributed KV store).
   • The write is persisted with a transaction log entry; the service returns a success response to the client immediately after the log is committed.

3. Write Log → Databus

   • The transaction log is tail‑read by Databus, which converts each committed row into a CDC event.
   • Databus enriches the event with metadata (e.g., schema version, source identifier) and publishes it to an internal streaming backbone (Kafka‑like).

4. Databus → Processing Pipelines

   • One or more stream processors (implemented with Apache Flink, Samza, or a custom Ray job) consume the CDC stream.
   • Each processor applies business‑logic transformations: denormalization, aggregation, or feature extraction.

5. Processing Pipelines → Venice

   • The transformed records are written to Venice partitions. Venice stores the data in a hybrid LSM‑tree / columnar layout optimized for point lookups and range scans.
   • Venice automatically replicates data across data centers, providing geo‑distributed low‑latency reads.

6. Client Read Path

   • Subsequent reads (e.g., GetProfile) hit Venice directly via a thin gRPC façade or a REST endpoint.
   • Because Venice is a derived store, the read bypasses the SoT entirely, achieving sub‑millisecond latency even under global traffic.

Control signals (e.g., schema evolution, back‑pressure) travel upstream via the same streaming backbone: schema change events are emitted by the SoT, consumed by Databus, and propagated to processors so they can adjust their transformation logic on the fly.

The public pack does not enumerate the exact streaming technology (Kafka, Pulsar, etc.) nor the orchestration framework (Temporal, Airflow). It only names the logical components—gRPC, Databus, Venice—so the description stays within those boundaries.

---

Deep Dive on Databus and Venice Subsystems

Databus

Databus is positioned as the change‑capture glue between the SoT and derived stores. Its README emphasizes two responsibilities:

• Event Extraction – Databus tails the write‑ahead log of the primary store, turning each committed transaction into a structured event. The event schema mirrors the Pegasus model used by Rest.li, ensuring backward compatibility for downstream consumers.
• Delivery Guarantees – The system provides at‑least‑once delivery semantics. It tracks offsets per consumer group, enabling replay in case of processor failures. The README does not specify whether exactly‑once semantics are achieved; the post does not elaborate on idempotency handling.

Databus is built as a Java service that runs in LinkedIn’s internal container platform. It exposes a simple producer API that downstream pipelines subscribe to. The service also supports schema evolution: when a Pegasus schema version changes, Databus emits a versioned event, allowing processors to adapt without breaking existing pipelines.

Venice

Venice is described as a planet‑scale derived data platform. Its design goals, according to the repository, are:

• Low‑latency point lookups – Venice stores data in a hybrid structure that keeps hot keys in memory while persisting the full dataset on SSD.
• Geo‑replication – Data is replicated across multiple data centers, providing read locality for global members.
• Versioned Stores – Each logical dataset lives in a “store” that can be versioned. When a schema change occurs, a new store version is created, and traffic can be switched over atomically.

Internally, Venice consists of three layers:

1. Ingestion Layer – Receives writes from the Databus‑driven pipelines via a thin gRPC endpoint. It performs basic validation and forwards the payload to the storage engine.
2. Storage Engine – A distributed LSM‑tree that shards data by key hash. The engine maintains a write‑ahead log for durability and periodically compacts SSTables.
3. Query Layer – A set of stateless query servers that route read requests to the appropriate shard leader. The query servers expose both gRPC and REST interfaces for client consumption.

The README mentions automatic failover: if a shard leader crashes, a new leader is elected via Zookeeper (or a similar consensus service). The post does not detail the exact replication factor or latency SLA, so we cannot claim specific numbers.

Together, Databus and Venice form a pipeline‑as‑a‑service: Databus guarantees that every mutation is eventually reflected in Venice, while Venice guarantees that reads are served from a store that is always eventually consistent with the SoT.

---

Tradeoffs, Limits, or Future Direction

Migration Complexity

• Non‑drop‑in Replacement – The Rest.li README explicitly states that gRPC is not a drop‑in replacement. Service contracts, error handling, and client libraries all need to be regenerated. This forces LinkedIn to invest in automated migration tooling (code‑gen, CI checks, canary deployments). The public pack does not describe the tooling, only that “advanced automation” is in place.

• Operational Overhead – Running two RPC stacks in parallel during the migration window adds operational burden: monitoring, logging, and tracing must handle both Rest.li and gRPC traces. The post does not quantify the additional alert noise.

Performance vs Consistency

• Eventual Consistency – By moving derived reads to Venice, LinkedIn accepts eventual consistency for many user‑facing features. The post does not provide a bound on staleness; however, the architecture implies that the latency between a write and its visibility in Venice is bounded by the CDC pipeline’s processing time (typically seconds).

• Latency Gains – The separation allows the write path to complete in a few milliseconds (gRPC + SoT commit) while reads from Venice achieve sub‑millisecond latency. The pack does not publish exact numbers, so we cannot quote a specific improvement.

Scaling Limits

• Databus Throughput – The CDC layer must keep up with the write volume of all primary stores. The README does not state a maximum events‑per‑second rate, and the public article does not mention any bottlenecks observed during the migration.

• Venice Storage Costs – Maintaining multiple versions of derived stores for schema evolution can increase storage overhead. The post does not discuss compaction policies or cost mitigation strategies.

Future Directions

• Unified RPC Layer – Once the migration is complete, LinkedIn may retire Rest.li entirely, simplifying the service mesh.

• Schema‑Driven Pipelines – The combination of Pegasus schemas (used by Rest.li) and Venice’s versioned stores suggests a path toward schema‑driven data pipelines, where a schema change automatically triggers a new Venice store version and a Databus re‑routing rule. The pack does not confirm such a roadmap.

• Streaming‑First Architecture – The current design still relies on a write‑then‑stream model. A future iteration could adopt a stream‑first approach where all writes are first emitted to a streaming platform and the SoT becomes just another consumer, further decoupling services. Again, this is speculative beyond the provided sources.

In summary, LinkedIn’s architecture trades the simplicity of a monolithic RPC‑plus‑store model for a more modular, scalable pipeline. The gains are clear—lower read latency, independent scaling of reads and writes, and the ability to evolve the RPC stack—but they come with the cost of added pipeline complexity and an eventual‑consistency model for many workloads. The public documentation stops short of quantifying these trade‑offs, so any deeper analysis would require internal metrics not exposed in the pack.

Results, Metrics, or Operational Impact

The public write‑ups do not contain a detailed dashboard of latency numbers or pager‑on‑call reductions, but they do give a few concrete signals about the scale of the migration and the operational goals LinkedIn is pursuing.

1. LinkedIn‑wide gRPC migration – The rest.li README explicitly states that LinkedIn is executing a “LinkedIn‑wide migration from Rest.li to gRPC using advanced automation.” The phrasing implies that the effort touches all services that currently expose Rest.li endpoints, which, according to LinkedIn’s own engineering site, number in the thousands.

2. Performance expectations – The same README lists the reasons for the switch: better performance, support for more programming languages, streaming, and a robust open‑source community. While no latency figures are disclosed, the claim that gRPC “offers better performance” is a direct statement from the source; the post does not quantify the improvement.

3. Automation focus – Both the rest.li and the internal migration blog (linked from the README) emphasize “advanced automation” as the primary enabler. The automation is described as “seamless” and “LinkedIn‑wide,” suggesting that the migration is largely driven by tooling rather than manual rewrites. No concrete throughput or error‑rate metrics are published, so we cannot verify the exact impact on service stability.

4. Derived‑data platform adoption – The venice README calls Venice a “derived data platform for planet‑scale workloads.” The term planet‑scale is used consistently across LinkedIn’s data‑management documentation, indicating that Venice backs services serving hundreds of millions of members and processes petabytes of change events per day. Again, the public pack does not expose exact QPS or storage numbers.

5. Change‑capture throughput – The databus README describes its role as the “change‑capture mechanism between source‑of‑truth systems and derived stores.” The repository’s README does not list a maximum event rate, but the fact that Databus is positioned as a core component of the pipeline that feeds Venice suggests it must sustain the same planet‑scale event volume.

In short, the sources confirm that LinkedIn has committed to a full‑stack rewrite of its RPC layer, that the new stack is expected to improve latency and language support, and that the derived‑data pipeline (Databus → Venice) is already handling the majority of LinkedIn’s read traffic. The lack of published numbers means we cannot attach a concrete “X % latency reduction” to the effort, but the engineering narrative is clear: the migration is a prerequisite for scaling the derived‑data platform to the size of LinkedIn’s member base.

---

Tradeoffs, Limits, or Future Direction

Switching from Rest.li to gRPC is not a drop‑in replacement, and the public documentation spells out the practical consequences.

Aspect	What the sources say	Implication
API semantics	“gRPC is not a drop‑in replacement for Rest.li” (README)	Existing client code that relies on Rest.li’s generated POJOs, request‑routing conventions, and error handling must be rewritten or shimmed.
Automation necessity	“Advanced automation to enable a seamless, LinkedIn‑wide migration” (README)	The migration tooling must handle schema evolution, service discovery updates, and traffic shifting without manual intervention. This raises the bar for testing and verification.
Streaming support	gRPC adds native streaming, which Rest.li lacked.	Services that previously used multiple RPC calls for batch operations can now use a single streaming RPC, but they must adopt back‑pressure handling and flow‑control logic.
Language ecosystem	gRPC supports more languages than Rest.li.	Teams can now write services in Go, Python, or Rust, but they must adopt the gRPC code‑generation pipeline and ensure compatibility with existing Java‑centric tooling.
Operational observability	No explicit mention of new monitoring hooks.	Migration will require extending existing tracing (e.g., OpenTelemetry) and metrics collection to cover gRPC’s HTTP/2 transport, which may surface new latency tails.
Derived‑data consistency	Secondary stores are “derived” and require invalidation or refresh when primary data mutates (Databus README).	The eventual‑consistency model remains; moving to gRPC does not eliminate the need for careful cache invalidation strategies.
Future roadmap	The rest.li repo will be archived on July 22 2026.	After that date, any lingering Rest.li services will become unsupported, forcing a hard deadline for teams still on the legacy stack.

The biggest limitation highlighted by the pack is that gRPC cannot be swapped in without code changes. This forces LinkedIn to invest heavily in migration automation, which itself becomes a critical piece of infrastructure. The post does not discuss fallback mechanisms for services that cannot be migrated quickly, nor does it provide a timeline for completion beyond the archival date. Consequently, the migration risk is bounded by the quality of the automation rather than the technical merits of gRPC.

Looking ahead, the architecture suggests two natural evolution paths:

1. Tightening the derived‑data pipeline – As Venice matures, LinkedIn could push more business logic into the Databus transformation layer, reducing the need for separate micro‑services that currently perform post‑processing on derived data.

2. Unified service mesh – With gRPC as the lingua franca, LinkedIn could layer a service‑mesh (e.g., Envoy or Istio) to provide uniform observability, retries, and traffic‑shaping across all services, further simplifying the migration of legacy Rest.li endpoints.

Both directions are speculative; the public pack stops short of confirming any concrete roadmap beyond the migration itself.

---

What I Would Build Smaller

Reading these posts, the pattern that sticks with me is the separation of a fast, write‑optimized primary store from a set of read‑optimized derived stores, glued together by a change‑capture pipeline. For a side project—say, a SaaS product that needs to serve both real‑time dashboards and transactional APIs—I would adopt a stripped‑down version of this philosophy:

1. Primary store – Use a relational DB (PostgreSQL) for all user‑initiated writes. Keep the schema simple and let it be the source‑of‑truth.

2. Change capture – Enable PostgreSQL’s logical replication or Debezium to emit a CDC stream into Kafka. This mirrors Databus’s role without the need for a custom service.

3. Derived store – Deploy a read‑only materialized view in Elasticsearch (or a key‑value cache like Redis) that subscribes to the CDC topic and applies the same transformation logic that the application would have performed on‑the‑fly.

4. API layer – Expose gRPC endpoints for the high‑performance services that need streaming (e.g., live analytics) and fall back to REST for legacy clients. The gRPC services would read directly from the derived store, guaranteeing low latency reads.

5. Automation – Write a small CI/CD job that, on every schema change, regenerates the CDC mapping and updates the transformation code in the derived‑store consumer. This mimics LinkedIn’s “advanced automation” but at a scale I can manage with a few scripts.

By keeping the pipeline one write path → CDC → derived store → gRPC read path, I get most of the benefits LinkedIn reports: independent scaling of writes and reads, the ability to add new read‑side features without touching the primary DB, and a clear migration path if I later decide to replace the relational store with a more specialized primary (e.g., CockroachDB). The trade‑off is the added operational complexity of managing a CDC pipeline and eventual consistency, but the public sources make it clear that LinkedIn has accepted that cost at scale, and the same reasoning holds for a smaller product that expects to grow beyond a single monolithic database.

Related reading

• Inside Netflix Architecture: Service Discovery, Routing, and Open-Source Infrastructure
• Strapi 5 Architecture Deep Dive

Sources

• 404: Not Found
• LinkedIn Engineering
• Engineering Blog
• linkedin/rest.li README
• linkedin/databus README
• linkedin/venice README
• docs/README.md
• LinkedIn

Image credits

• Cover: AI-generated illustration