Engineering · 2026-04-13

Circuit Breakers for Identity Pipelines: Graceful Degradation When a Carrier API Goes Dark

By J. W. Bouckaert

The outage you don't see coming

Every identity verification system that depends on carrier APIs has an implicit assumption baked into its architecture: the carrier API will respond. Not eventually—now. Within the latency budget. With accurate data.

This assumption fails regularly.

Carrier APIs are federations of aggregator relationships, SS7/Diameter gateway contracts, and country-specific routing tables. When Twilio Lookup v2 returns a SIM swap status for a UK mobile number, that response traversed at least three intermediaries before reaching your application. When Vonage Number Insight Advanced returns roaming status for a German MSISDN, the signal path likely touched a Diameter gateway in Frankfurt, an HLR in Düsseldorf, and a response aggregation layer in London.

Any link in that chain can fail. And when it does, your identity pipeline has a decision to make.

A carrier API will go down. The useful question is what your identity pipeline does in the 47 minutes before it comes back up.

Most identity platforms handle this with a timeout and a retry, which amounts to hope with a configurable interval.

Limits of timeouts and retries

The standard approach to carrier API resilience looks like this:

Request → Timeout (5s) → Retry → Timeout (5s) → Retry → Fail → Return error

This pattern has three problems that compound during real outages:

1. Latency accumulation

If your per-provider timeout is 5 seconds and you retry twice, a single failing provider adds 15 seconds of latency to every verification request. For an identity platform that answers inside the request, 15 seconds is a service outage wearing a timeout's clothing.

2. The retry storm

When a carrier API is struggling under load, retries from every client make the problem worse. If 10,000 concurrent verifications each retry twice against a degraded provider, the provider receives 30,000 requests instead of 10,000. This is the retry storm antipattern—your resilience mechanism becomes a denial-of-service amplifier.

3. Binary failure mode

Timeout-and-retry produces a binary outcome: success or failure. There is no intermediate state. Your system either got a carrier signal or it didn't. For a multi-provider signal fusion architecture that weights and correlates signals from multiple carriers, this binary outcome discards the nuance that makes fusion valuable.

The circuit breaker pattern

The circuit breaker pattern, formalized by Michael Nygard in Release It! (2007) and adopted across distributed systems since, solves all three problems by shifting failure detection from the request level to the provider level.

A circuit breaker monitors the health of an upstream dependency over a rolling window of requests. When failures exceed a threshold, the breaker opens—blocking all requests to that provider without waiting for timeouts. After a cooldown period, the breaker transitions to half-open and probes the provider with a single request. If the probe succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker reopens.

State machine

This is not a new pattern. Netflix's Hystrix popularized it for microservices. Polly brought it to .NET. Resilience4j is the current Java standard. What is new is applying it to identity verification pipelines where the upstream dependencies are carrier APIs with fundamentally different failure characteristics than typical microservices.

PasskeyBridge's circuit breaker implementation

PasskeyBridge implements a per-provider circuit breaker for every carrier API in the signal fusion pipeline. Each provider—Twilio, Vonage, and direct HLR/SS7 probes—has an independent breaker with configurable thresholds.

Configuration parameters

The sliding window

The failure threshold uses a sliding window rather than a simple counter. This matters because carrier API failures are often bursty—a routing change at an aggregator can cause a 10-second burst of errors followed by full recovery. A simple counter would open the breaker on the burst. A sliding window with a rate-based threshold (e.g., >50% failure rate over 60 seconds with a minimum of 10 requests) distinguishes between transient bursts and sustained outages.

// Simplified sliding window implementation
interface ProviderHealthWindow {
  provider: string;
  windowMs: number;
  results: { timestamp: number; success: boolean; latencyMs: number }[];
  state: 'closed' | 'open' | 'half-open';
  lastStateChange: number;
}

function evaluateCircuit(window: ProviderHealthWindow): 'closed' | 'open' {
  const now = Date.now();
  const recent = window.results.filter(r => now - r.timestamp < window.windowMs);

  if (recent.length < 10) return 'closed'; // insufficient data

  const failureRate = recent.filter(r => !r.success).length / recent.length;
  const p95Latency = percentile(recent.map(r => r.latencyMs), 95);

  if (failureRate > 0.5 || p95Latency > 3000) return 'open';
  return 'closed';
}

Latency as a failure signal

A carrier API that responds in 8 seconds with a valid result is, from the identity pipeline's perspective, functionally equivalent to one that doesn't respond at all. PasskeyBridge treats latency breaches as failures within the circuit breaker's sliding window. If a provider's P95 latency exceeds 3,000ms over the observation window, the breaker opens even if every response was technically successful.

This is critical for identity verification because verification latency directly impacts conversion rates. A checkout flow that adds 8 seconds of identity verification latency is a checkout flow that loses customers.

Fallback signal weighting

Opening a circuit breaker is half the problem. The other half is: what happens to the verification request that was going to use that provider's signal?

In a single-provider architecture, the answer is failure. In PasskeyBridge's multi-carrier signal fusion architecture, the answer is dynamic weight redistribution.

Normal operation weights

During normal operation, each provider contributes a weighted signal to the composite risk score:

These default weights are adjusted per-country, per-carrier based on historical accuracy. Twilio's weight for US T-Mobile numbers might be 0.45 because of strong direct carrier relationships, while its weight for Nigerian MTN numbers might be 0.30 because of aggregator-dependent coverage.

Weight redistribution during outage

When a provider's circuit breaker opens, its weight drops to zero and the remaining providers are renormalized:

Example: Twilio breaker opens

Example: Vonage and HLR breakers open (worst case)

This renormalization happens per-request rather than as a global configuration change. The moment a breaker closes, the provider's weight returns to its normal value on the next request.

The minimum trust threshold

Renormalization alone is not sufficient. A single provider carrying 100% of the weight produces a verification decision with significantly lower confidence than a three-provider fusion. PasskeyBridge enforces a minimum trust threshold to prevent low-confidence verdicts from being issued as if they were high-confidence:

Composite Score = Σ(provider_weight × provider_signal_score)

If active_providers < required_minimum:
    return { status: "insufficient_signal", confidence: composite_score }

If composite_score < minimum_trust_threshold:
    return { status: "low_confidence", confidence: composite_score }

The default minimum trust threshold is 0.6 (configurable per tenant). When the composite score falls below this threshold—which becomes more likely as providers go offline—the system returns an explicit insufficient_signal status rather than a binary pass/fail. This gives the consuming application the information it needs to make its own risk decision: proceed with step-up authentication, queue the request for manual review, or reject.

The half-open probe strategy

The half-open state is where circuit breakers succeed or fail. A naive implementation sends a burst of traffic to a recovering provider, potentially knocking it back down. PasskeyBridge's half-open strategy avoids this with three mechanisms:

1. Single-request probing

When the cooldown timer expires, the breaker transitions to half-open and forwards exactly one real verification request to the provider. The probe is a real request from a real user's verification flow. Using real traffic ensures the probe exercises the full provider path, including country-specific routing and aggregator dependencies that a synthetic ping would miss.

2. Graduated recovery

A single successful probe is not sufficient to close the breaker. PasskeyBridge requires two consecutive successes (configurable via success_threshold) before transitioning from half-open to closed. This prevents premature closure on a provider that is oscillating between healthy and unhealthy states—a common pattern during carrier API maintenance windows.

3. Exponential backoff on repeated opens

If a provider's breaker opens, closes after half-open probing, and then opens again within a short window, the cooldown period doubles:

This backoff prevents the circuit breaker from rapidly cycling between open and closed during a prolonged partial outage, which would create unpredictable latency for verification requests.

Observability and alerting

A circuit breaker that operates silently is a circuit breaker that creates surprises. PasskeyBridge instruments every state transition:

Metrics emitted

These metrics feed into PasskeyBridge's observability dashboard and can trigger alert rules via configured channels (Slack, PagerDuty, OpsGenie, webhook).

Audit trail integration

Every circuit breaker state change is recorded in the append-only audit log with:

• Provider identifier
• Previous and new state
• Failure rate at time of transition
• P95 latency at time of transition
• Active verification count at time of transition

This audit trail satisfies SOC 2 Type II requirements for change management and incident documentation. When an auditor asks "what happened during the Twilio outage on March 15th," the circuit breaker's state transitions provide a millisecond-accurate timeline.

Real-world scenario: The Vonage routing incident

To illustrate how these mechanisms work together, consider a realistic incident scenario:

T+0s: Vonage Number Insight responses for German MSISDNs begin returning 503 errors. PasskeyBridge's sliding window starts accumulating failures.

T+12s: Vonage's failure rate exceeds 50% over the 60-second window (6 failures out of 11 requests). The circuit breaker opens. All subsequent verification requests skip Vonage entirely.

T+12s (same instant): Signal weights are redistributed. Twilio absorbs 0.53 weight (up from 0.40). HLR probes absorb 0.47 weight (up from 0.25). Verification continues with two providers.

T+12s to T+42s: 847 verification requests are processed without any Vonage calls. Average verification latency drops from 142ms to 98ms (because the slowest provider was Vonage). No requests fall below the minimum trust threshold because two providers still exceed the 0.6 floor.

T+42s: The 30-second cooldown expires. The breaker transitions to half-open. The next verification request for a German MSISDN is forwarded to Vonage as a probe.

T+42s: The probe returns a 503. The breaker reopens. Cooldown resets to 60 seconds (exponential backoff, 2nd cycle).

T+102s: Second half-open probe. Vonage returns a valid response in 340ms. Probe #1 succeeds. The breaker remains in half-open, waiting for a second consecutive success.

T+103s: Next German MSISDN request is probed. Vonage returns valid data in 280ms. Probe #2 succeeds. The breaker closes. Vonage's weight returns to 0.35.

Total impact: Zero verification failures. 847 requests processed with degraded but sufficient signal coverage. Latency actually improved during the outage. The entire incident was invisible to end users.

Carrier APIs versus microservices

One nuance that distinguishes carrier API circuit breakers from typical microservice circuit breakers is that carrier APIs have geography-dependent failure modes.

A Twilio Lookup request for a US number routes through different infrastructure than a request for a Kenyan number. Vonage's Number Insight for UK MSISDNs uses different aggregator relationships than for Brazilian numbers. A provider can be fully operational for one country while completely down for another.

PasskeyBridge addresses this with per-country circuit breakers. Each provider has a separate breaker for each country code:

twilio:US → closed
twilio:GB → closed
twilio:DE → open (aggregator issue)
twilio:NG → half-open (recovering)
vonage:US → closed
vonage:GB → closed
vonage:DE → closed
hlr:US → closed
hlr:GB → closed
hlr:DE → closed

This granularity ensures that a Twilio routing issue for German numbers doesn't affect verification for US numbers, even though both use the same Twilio API endpoint.

Cost of omitting circuit breakers

The alternative to circuit breakers is what most identity platforms do today: hope for the best and retry when things break.

The cost is measurable:

The last row is important. When all providers are down, a circuit breaker doesn't create a miracle—there is no signal to fuse. But it does fail fast and explicitly. The consuming application receives an insufficient_signal status in <1ms instead of waiting 15 seconds for three serial timeouts before getting an opaque error.

Implementation considerations

Thread safety

Circuit breaker state must be safe for concurrent access. PasskeyBridge's implementation uses atomic operations for state transitions and lock-free data structures for the sliding window. In a Deno Deploy edge function environment, this means using Atomics for state flags and a pre-allocated ring buffer for the sliding window.

State persistence

Circuit breaker state is ephemeral by design. If a Deno Deploy isolate restarts, the breaker resets to closed. This is intentional—a fresh isolate should discover provider health through its own observations rather than inheriting stale state from a previous instance. The sliding window fills within seconds of normal traffic, so the "cold start" penalty is minimal.

Testing

PasskeyBridge's CI pipeline includes chaos testing for circuit breakers. The test suite injects controlled failures into provider mocks and verifies:

• Breaker opens at the configured threshold
• Weight redistribution is mathematically correct
• Half-open probes follow the configured schedule
• Exponential backoff increases cooldown correctly
• Minimum trust threshold prevents low-confidence verdicts

These tests run on every deployment. A regression in circuit breaker behavior is a blocking failure.

Summary

Carrier APIs will fail. The question is whether your identity pipeline treats those failures as exceptional events that produce errors, or as expected conditions that trigger graceful degradation.

Circuit breakers transform carrier API failures from identity pipeline outages into observability events. They protect failing providers from retry storms, protect your users from accumulated latency, and protect your verification accuracy through dynamic signal weight redistribution.

The architecture is simple: three states, a sliding window, and a weight redistribution function. But the difference between having it and not having it is the difference between a 30-minute outage and a 12-second, invisible failover.

PasskeyBridge's circuit breakers are active on every verification request, for every provider, in every country. Nothing has to be enabled during an incident; they are the reason you don't have incidents.

Explore the multi-carrier signal fusion architecture →

See how in-request playbook execution closes the gap →

Start your free trial →

Start free · Test the API