Cosmos DB Architecture (4-Layer Path)

When you call await container.ReadItemAsync(id, partitionKey), what actually happens? A surprising amount, distributed across four layers each running on different hardware, often in different physical buildings. Each layer adds latency and capability; understanding the path makes operational debugging tractable.

This lesson maps the request flow end-to-end.

The big picture

   ┌─────────────────────────────────────────────────────────────┐
   │  Your App  (CosmosClient SDK, holds routing cache)          │
   └────────────────────┬────────────────────────────────────────┘
                        │
              Gateway mode  or  Direct mode (TCP)
                        │            │
                        ▼            │
   ┌─────────────────────────┐       │
   │  Layer 1: Gateway       │       │
   │  Auth · Routing · Retry │       │
   └─────────┬───────────────┘       │
             │                       │
             ▼                       ▼
   ┌─────────────────────────────────────────────┐
   │  Layer 2: Federation                        │
   │  Account → Database → Container → Partition │
   └─────────────────────┬───────────────────────┘
                         │ "go to physical partition X"
                         ▼
   ┌─────────────────────────────────────────────┐
   │  Layer 3: Partition Manager                 │
   │  4-replica set · Paxos · primary + 3 sec.   │
   └─────────────────────┬───────────────────────┘
                         │ "do the read/write"
                         ▼
   ┌─────────────────────────────────────────────┐
   │  Layer 4: Storage Engine                    │
   │  Bw-tree · index · page cache · WAL         │
   └─────────────────────────────────────────────┘

Each layer has a clear, separable responsibility. Each can fail independently and degrades the system in a recognizable way.

Layer 1: The Gateway

The Gateway is the public face of Cosmos. Every account has a regional gateway service (<your-account>.documents.azure.com) that handles:

• Authentication — validates the master key signature or AAD token on every request.
• Authorization — applies RBAC role assignments.
• Routing — determines which physical partition a request belongs to (queries the federation layer for the routing table).
• Region selection — for multi-region accounts, picks the closest write or read region.
• Retry logic — handles transient faults inside the cluster.

In Gateway mode, your SDK talks only to the Gateway — every request is HTTPS POST to documents.azure.com. The Gateway then forwards to the right partition via internal TCP.

In Direct mode, your SDK still talks to the Gateway initially (for connection setup and routing-cache refresh), but actual reads and writes go straight to partition replicas via TCP. The Gateway becomes a sideline service for metadata only.

Latency cost — Gateway mode adds 5–10 ms per request because every operation does an extra hop. Direct mode skips that hop.

Layer 2: Federation

A “federation” is Cosmos’s term for a cluster of physical machines that host many partitions across many tenants. The Federation layer maintains the global routing table:

Account "myapp"
  Database "prod"
    Container "users"
      Partition Key Range [00..00 → 7F..FF]  →  Federation 12, Replica Set 5482
      Partition Key Range [80..00 → FF..FF]  →  Federation 12, Replica Set 5483
    Container "orders"
      ...

When a partition splits, the Federation layer updates the table; SDK clients refresh their cache the next time they hit a stale entry. The Federation layer also coordinates partition migration when one federation runs out of capacity.

You never see this layer. It’s an internal control plane. The only time it matters is during very rare events like tenant migration, where you might see a brief routing-cache stale window.

Layer 3: Partition Manager (per-partition replica set)

Each partition is a replica set of 4 — one primary, three secondaries — running on different fault domains within a region. This layer:

• Runs the consensus protocol (Paxos-derived) to agree on the order of writes.
• Handles leader election when the primary fails.
• Coordinates replication of the write log to secondaries.
• Manages read load balancing across replicas (within consistency constraints).

A write here looks like:

1. SDK or Gateway sends write to the primary.
2. Primary writes to its WAL on local SSD (durable).
3. Primary replicates the WAL entry to the three secondaries.
4. Once 3 of 4 replicas (quorum) confirm, the write is committed.
5. Primary acks the SDK.

This entire dance typically takes 2–5 ms within a region. Most of that is replica round-trips; the local SSD write is microseconds.

The replica set is what gives Cosmos its durability + availability guarantees. Lose one machine — the cluster keeps writing. Lose the primary — a new one is elected from the secondaries in seconds. Lose two of four — writes block until a replica recovers (rare but possible in major cloud incidents).

Layer 4: Storage Engine

Inside each replica is the actual on-disk data. This is the Bw-tree we discussed in F2:

• Per-container Bw-tree for the data.
• Per-indexed-path Bw-tree for the index.
• WAL on local SSD for durability before commit.
• Page cache in RAM for hot pages.

When a read arrives, the storage engine:

1. Looks up the document in the Bw-tree (1–2 page reads, often cache hits).
2. Applies any pending deltas (lock-free read of the chain).
3. Returns the materialized document.

When a write arrives (after consensus has committed it):

1. Appends the WAL entry to a local commit log.
2. Adds a delta to the Bw-tree’s mapping table via CAS.
3. Updates each affected index’s Bw-tree similarly.
4. Acks “durable” back up to the partition manager.

The storage engine is also where indexing policies apply (V07) — every JSON path in the indexing policy is its own Bw-tree, and every write ripples through all of them.

Reading the layer in error messages

When something goes wrong, you can usually tell which layer is to blame from the symptoms:

The SDK exposes diagnostic info that pinpoints the layer — CosmosDiagnostics in .NET, getDiagnostics() in Java/Node — and printing it on slow queries is one of the highest-leverage habits an experienced Cosmos engineer develops.

What changes for multi-region

In a multi-region account, each region has its own full stack — its own Gateway, Federation, Partition Manager, and Storage. The replication that crosses regions is async (for session and weaker consistency) or sync (for strong consistency).

The SDK’s PreferredLocations setting determines which region’s Gateway you talk to first. Reads can be local; writes go to the write-region (single-master) or the closest region (multi-master).

V12 (Global Distribution) drills into the cross-region propagation in detail.

Where this connects

• F1 Replication — Layer 3 (the replica set) is the consensus + replication. F1 explains how 4 replicas stay in sync.
• F2 Storage Engines — Layer 4 is the Bw-tree. F2 explains why it’s an append-friendly tree.
• F3 Partitioning — Layer 2’s routing table is built on the hash-range scheme F3 describes.
• V08 SDK Best Practices — most production tuning happens at Layer 1’s SDK config (singleton client, connection mode, retry policy).
• V12 Global Distribution — what cross-region propagation looks like, and the trade-offs.

Once the four-layer model is in your head, every Cosmos issue points at a specific layer. That’s the entire point of building this mental model — diagnosis becomes mechanical.