Engineering · 2026-07-17

Multi-Region Active-Active Identity: Conflict-Free Replicated Trust State Across Three Continents

By J. W. Bouckaert

The failure mode nobody ships

Every identity vendor at scale eventually ships the same architecture diagram: a primary region, a warm standby, and a promise about failover. Then a fibre cut severs the two, the standby lags by ninety seconds, and a revoked credential clears authentication in every region that has not caught up yet. The post-mortem always contains the same sentence: replication was asynchronous.

Asynchronous replication was not the bug. Treating a single-writer database as the source of truth for a globally-distributed trust decision was the bug. The moment identity latency budgets fell below the speed of light between continents—and for us that happened the year we committed to answering inside the request—single-primary architectures stopped being an option.

This article describes the architecture we run instead: a three-region active-active identity backplane in which every region accepts writes, no region is authoritative, and the state that matters most for security—revocation lists, session invalidations, and replay-protection nonces—converges under a delta-state Conflict-Free Replicated Data Type (CRDT) discipline. The counter-intuitive claim we will defend is that eventual consistency is the safer choice for revocation than strong consistency, provided you get the direction of the eventuality right.

The cost of active-active

Textbook active-active is easy to draw and hard to reason about. Three regions—call them us-east, eu-west, and ap-southeast—each accept full read and write traffic. A user in Frankfurt hits eu-west, a session token minted there is presented ten seconds later in Singapore, and the verifier in ap-southeast must decide whether that token is (a) valid, (b) revoked, or (c) unknown-because-not-replicated-yet.

The interesting distinction is between (b) and (c). Any system that conflates them ships a security incident.

Physical constraints set the floor. Round-trip time between the three regions we operate is:

Under a strong-consistency model (Paxos, Raft, Spanner-style TrueTime) a write must reach a quorum before returning. For a three-region deployment, that means the nearest two regions must acknowledge—so an ap-southeast write is bound by the RTT to eu-west and back. A revocation issued in Singapore takes at least one full 168 ms round trip to commit. A verifier in us-east that consulted ap-southeast during that window would get the pre-revocation state.

Under an eventual-consistency model the write returns instantly in the region where it originated and propagates asynchronously. A revocation issued in ap-southeast is durable locally the moment it commits, and us-east learns about it milliseconds to seconds later.

The safety question is which of those two failure modes is worse. Under strong consistency, a revoked credential can be accepted anywhere in the world for up to one RTT, because the revocation has not yet committed. Under eventual consistency with the right CRDT choice, a revoked credential cannot be accepted in the region where it was issued (local read-your-writes), and can be accepted in a remote region only for the duration of the propagation gap. That gap is measurable, boundable, and—critically—can be closed by a verifier that is willing to spend one RTT of its own to double-check across regions.

That last property is what makes eventual consistency the safer choice: the verifier retains agency. Under strong consistency the verifier has none; it consults its local replica and trusts the quorum. If the quorum is stale because a fibre cut turned it into a minority, the verifier does not know.

CRDTs, and which ones

A Conflict-Free Replicated Data Type is a data structure whose merge function is commutative, associative, and idempotent. Replicas may receive updates in any order, any number of times, and still converge to the same state. The original taxonomy is Shapiro et al., 2011, and the delta-state refinement we lean on is Almeida, Shoker, and Baquero, 2018.

For an identity backplane, three CRDT primitives carry almost the entire load:

Revocation is the load-bearing case, and it happens to be the easiest. A revocation is monotonic in exactly the direction we need: once revoked, a credential must never become un-revoked without an explicit re-issuance, which is a new credential with a new identifier. The operation "add credential ID c to the revocation set" is safe to apply any number of times, in any order, across any number of regions. That is the OR-Set contract, and it is why revocation is a natural fit for CRDT replication—more natural than for the strong-consistency architectures the industry defaulted to a decade ago.

Contrast this with a hypothetical "credential status" register that could flip revoked → active → revoked. The moment a state can move backwards, ordering matters, and the CAP-theorem tax becomes real. The W3C StatusList2021 specification made this observation explicit at the credential layer; we make the same observation at the replication layer.

Delta-state instead of full-state

Naïve state-based CRDTs (CvRDTs) replicate by shipping the entire state on every anti-entropy round. For a revocation set that grows monotonically across a fleet of tenants, that state is unbounded in the long run. Delta-state CRDTs solve this: each replica keeps a delta buffer of recently-applied mutations and gossips only those, falling back to full-state exchange only during initial sync or after prolonged partitions.

The wire economy on our production backplane, measured over the last 30 days:

A revocation issued in eu-west is visible in us-east in under a second at the 99th percentile, and in ap-southeast within roughly two seconds. Compare that to the ninety-second lag of a warm-standby architecture, or the OCSP freshness windows that permit up to seven days of stale revocation data by default in traditional PKI.

The anti-entropy layer

A CRDT tells you what to merge. It does not tell you when or between whom. That is the job of the anti-entropy layer, and here we borrow directly from the Amazon Dynamo and Riak lineage: gossip plus Merkle trees.

        us-east
         /    \
        /      \
   eu-west ── ap-southeast
     (250ms gossip round, all-pairs)

Each region maintains a Merkle tree over its OR-Set. Every gossip round, the three regions exchange root hashes. When two roots disagree, the diverging subtree is walked to identify the specific keys that need reconciliation, and only the delta buffer for those keys is shipped. This is O(log n) in the number of keys under partial disagreement, and it keeps the wire cost proportional to the rate of new revocations, not the size of the revocation history.

Merkle trees also solve the Byzantine problem: the gossip payload is signed by the originating region under its hybrid signature suite, and a receiving region rejects any purported delta whose signature does not verify against the sender's registered key. A compromised replica cannot inject a fake revocation into the global set, and it cannot un-revoke anything, because the underlying OR-Set does not permit un-observed removes.

Replay protection

Revocation is easy because it is monotonic. Replay protection is hard because it is bounded but grow-only: every nonce a verifier consumes must be remembered for the token's TTL, and forgotten thereafter. This is the case where the CRDT literature usually reaches for a Delta-Buffer with Causal Delivery, and we do the same, with two adaptations.

First, the nonce set is sharded by TTL bucket. A nonce with a 15-minute TTL lives in a 15-minute-window OR-Set that is garbage-collected as a whole when the window expires. The tombstones the OR-Set would normally accumulate are collapsed at window-boundary time, keeping steady-state memory bounded.

Second, replay-protection is a case where cross-region ordering matters within a token's lifetime. A one-time-use nonce presented in us-east and then, 40 ms later, in eu-west must be rejected in the second region. Delta-CRDT convergence for this case runs on a tighter cross-region path: rather than 250 ms gossip, nonce consumption is pushed on a best-effort direct link with a 15 ms target. If the push has not landed by the time the second region evaluates, the verifier in eu-west does the belt-and-braces thing—it consults us-east synchronously for that specific nonce. This is the "verifier retains agency" property in practice: a verifier that cares about a specific edge case can pay one RTT of latency to close the eventual-consistency gap for that decision alone.

The math works because nonce presentation is a rare cross-region event. Roughly 2% of our replay-protection lookups traverse regions in the same window; the other 98% are same-region reads served from the local OR-Set in under 5 ms.

The three failure modes we explicitly design for

Regional partition. A fibre cut isolates ap-southeast from the other two regions for six hours. Under our model: writes continue in all three regions independently. Revocations issued in ap-southeast accumulate in its delta buffer. When the partition heals, the anti-entropy layer performs a full Merkle-tree walk against the other two regions and replays every buffered delta. Convergence completes in under 90 seconds for a six-hour partition with a working-day revocation load. No writes are lost. No revocations are silently missed.

Split-brain writes. During the partition, tenant T has credential c revoked in us-east and simultaneously re-issued in ap-southeast under a new credential ID c'. When the partition heals, the OR-Set contains c (revoked) and c' (active). No conflict exists because the operations were on different keys, and the semantic invariant—"a credential is revoked once anyone anywhere revokes it"—holds without any coordination. The zero-PII architecture means neither c nor c' leaks the underlying subject; both are hashed identifiers.

Byzantine replica. A compromised region attempts to un-revoke a credential by gossiping a delta claiming the revocation was never observed. This is structurally impossible in an OR-Set: a remove operation is valid only against a specific add-tag that must be quoted verbatim, and the operation must be signed by the originating region. A region cannot forge a remove for an add-tag it did not originate. Even under full compromise of a single region, the other two regions' local state remains authoritative for their originating adds. The worst-case impact is that the compromised region's own adds cannot be trusted—which is the same guarantee any competent cross-region trust model gives.

CAP, revisited for identity

The CAP theorem says a distributed system can offer at most two of Consistency, Availability, and Partition Tolerance during a network partition. For identity, the choice looks like this:

Framed this way, the choice is not close. A CP system optimizes for not making a bad decision quickly. An AP system optimizes for making the correct decision somewhere, immediately. In every incident scenario we have seen or modelled—stolen device, SIM swap, insider compromise, a wave of fraudulent carrier signals—the response is a race between the defender's revocation and the attacker's next authentication attempt. AP wins that race in the region where the revocation is issued. CP loses it everywhere until quorum.

This is the same reasoning Amazon documented for the Dynamo shopping-cart use case—"never lose a write"—applied to a security-critical domain. The Delay-Tolerant Identity architecture we published earlier is the same discipline applied to the lack of network at all; the CRDT backplane is what makes that discipline coherent across regions when the network is present but slow.

Limits of the backplane

CRDTs are not a replacement for correctness. Two things they explicitly do not give you:

1. Global ordering of unrelated operations. If your business logic requires a strict serialization across regions—say, "no two administrators may simultaneously modify the same tenant policy"—you need a separate coordination primitive. We use a per-tenant lease held in a single home region for these operations, at the cost of one RTT for cross-region policy edits. This trade-off is deliberate: policy edits are rare, revocations are hot.
2. Read-your-writes across regions. A revocation issued in us-east is guaranteed to be visible in us-east immediately, but a verifier in eu-west reading milliseconds later may miss it. Applications that need cross-region read-your-writes for a specific decision must fall through to the anti-entropy sync-on-read path—which is available, but costs a real RTT.

For the circuit-breaker paths in the identity pipeline, the sync-on-read is off by default and on for exactly the operations that warrant it. That decision lives in the verifier's policy.

Operational reality after twelve months

We committed to this architecture at the start of 2026 and have run it in production across three regions since April. Selected numbers from the first six months:

The 43-minute partition on 2026-05-19 was a Cloudflare peering issue between eu-west and the other two regions. All three regions continued to serve traffic. Revocations issued in eu-west during the outage were absorbed into a 12 MB delta buffer and replayed against us-east and ap-southeast in 71 seconds after the peering restored, with full Merkle-tree validation on every entry. The observability layer surfaced the divergence within 90 seconds of the fibre cut, giving operators enough runway to page a human before the partition mattered.

Downstream consequences

The interesting consequence of an active-active CRDT backplane goes beyond the resilience number: every downstream design decision that used to be gated on "which region is authoritative" simply stops being a question. Data residency policies become a routing decision. Edge verification becomes viable because the edge can trust a local read. Cold-start latency for serverless identity functions drops because there is no primary-region round trip on the hot path.

For a platform whose core promise is every signal verified, inside the request, the CRDT backplane is what makes the promise defensible under partition. It is the layer that lets three regions on three continents behave as one identity fabric without any of them being in charge.

The wire format is small. The merge function is proven. And the direction of eventuality—towards more revoked rather than less—matches the direction security wants to go anyway.

Read the StatusList2021 vs accumulators analysis → Explore event-sourced trust decisions → See the delay-tolerant identity architecture → Get started with PasskeyBridge →

Start free · Test the API