Engineering · 2026-08-24

Benchmarking an Identity Verification API: Latency, Freshness, Failure Injection

By J. W. Bouckaert

The demo is not your network

Every vendor of an identity verification API has a latency number. It is almost always measured under three conditions that are not yours: on the vendor's network, on a warmed instance, and at the median. The user who abandons a login is on none of those. They are on your network, they hit a cold path, and they live in the tail.

A benchmark exists to reproduce the conditions you will actually run under and to measure the things that decide whether an identity verification API is safe to put in front of a sign-in: how fast it answers, whether the answer is still true, and what it does when something breaks. Most evaluations measure the first badly and skip the other two entirely. This is the method we would use to evaluate a vendor, including us.

One scoping note before the method. This is about benchmarking a verification and signal API, the kind that takes a request and returns a signed decision. It is not about benchmarking an identity provider. If the thing you are testing owns user profiles, enrollment, and token issuance, you are measuring a different product with a different budget.

Measure from where you run, in percentiles

The single most common benchmarking error is reporting an average. An average latency of 40ms is compatible with one request in twenty taking 400ms, and that one request is a real person staring at a spinner during checkout. The tail is the experience. Report p50, p95, and p99, and treat the p95 as the number that matters.

Four conditions make a latency measurement honest:

• From your infrastructure. Measure from the region and network you will serve from. A vendor's demo harness reports on the vendor's network. A number collected inside the vendor's own cloud tells you about the vendor's cloud.
• Cold and warm, separately. The first request after an idle period pays initialization costs that steady-state traffic does not. Report both. The engineering behind that gap is its own subject; we covered it in cold-start latency for serverless identity functions.
• With a sample size. A p99 computed over 50 requests is noise dressed as a statistic. State how many requests the percentiles are drawn from and over what window.
• Against a real endpoint. A mocked response measures your mock.

We built a public surface to make the honest version of this trivial. The playground runs real requests from your browser with no signup and no API key: a single ping issues one real GET /health and times the round trip with the browser's own clock; a burst fires ten sequential pings and returns p50, p95, and p99 from your network; an echo mode shows exactly which request headers survive the trip to the edge. The header on that page states the constraint we hold ourselves to: "Real requests against the production edge proxy. No signup, no API key, no mocks. The latency you see is the latency you'll get."

What the playground measures is transport and edge round-trip from your machine. That is the floor under every real call, measured on your connection rather than ours, and it is the right first number. It is not the whole number, which is the next point.

Separate the legs you are timing

An identity verification API has no single latency, only a sum of legs, and a benchmark that reports one figure has usually hidden which legs it included. Decompose the path into three parts and time each one:

The distinction between the second and third legs is where most latency claims quietly overreach. We publish a measured figure for the decision path rather than a target: the database-backed trust decision that runs with no external upstream, such as a proof-of-possession check. It is recorded on every request as decision_ms and read back at p50, p95, and p99 against real request events drawn from production traffic.

A carrier-network lookup is a different leg with a different owner. Carrier signal is sourced via GSMA Open Gateway through carrier-grade providers (Vonage primary; Twilio and others via bring-your-own-key), and the provider's response time dominates that path. It is bounded by the carrier ecosystem, which sits outside our control. A benchmark that folds a carrier lookup into "the API's latency" is largely measuring the carrier. When you evaluate any vendor, make them name which leg their headline number covers. If they cannot, the number is marketing.

Transport time and decision time

Here is the candor that a benchmarking guide owes you, applied to our own numbers. We used to publish a headline figure of fifty milliseconds. The first end-to-end production measurement, in September 2026, put the decision path at p50 213ms, and we withdrew the figure rather than restate it at a new one. After folding four reads into a single round trip, the same probe reads p50 175ms and p95 209ms on production over 60 requests. Those are sandbox-key numbers, and a sandbox key pays one extra round trip worth about 68ms at p50, so the live-key path derives to roughly 98ms. The only direct live-key measurement we hold is on staging: p50 90ms over 30 requests. Every one of those figures moves with load, region, and which leg you happen to time, which is the reason the qualifiers matter more than the number does.

That is the whole lesson. A vendor who quotes one millisecond number with no percentile, no sample size, and no named scope is quoting a best case and inviting you to read it as a promise. The useful question is never "what is your latency." It is "what is your p95, over how many requests, for which operation, measured from where." An answer that survives those four qualifiers is worth something. An answer that evaporates under them was never a measurement.

Freshness

A fast answer to a stale question is worse than a slow answer to a current one, because it arrives with the authority of speed and the content of a guess. This is the dimension almost every identity benchmark omits, because a latency probe cannot see it.

In carrier-signal work we rank freshness above signal strength. A strong signal observed an hour ago tells you about an hour ago; if an account was taken over forty minutes ago, the strength of a stale reading is precisely how convincing the wrong answer will be. We made the full argument in why carrier signal freshness beats strength. For a benchmark, freshness turns into four questions you put to the vendor, because they are answered by architecture rather than by a stopwatch:

• Does every signal carry an observation timestamp, or only a value? A value with no time is a claim with no expiry.
• What does the API do with a signal older than its ceiling: reject it, downweight it, or serve it anyway and let you find out later?
• Is the freshness ceiling configurable per assurance level, so a high-value transaction can demand a tighter one?
• Is the response dated on the wire, so you can read the age of the answer you were given?

Our published position is a freshness-first fusion with a sub-second ceiling for any signal that contributes to a positive trust decision. The point for your evaluation is not that number, it is that a vendor should be able to state theirs in milliseconds and show where it is enforced. "Real-time" is not a number. Ask for the number.

Failure injection

Every identity API is fast and correct on the happy path in a controlled demo. No incident is ever the happy path. An API you have only tested while it was healthy is an API you have not evaluated, because the behavior that determines your blast radius only appears under stress.

There is rarely a fault-injection switch to flip, and there should not be a public one; ours has none, because an endpoint that lets any caller induce failures is itself a liability. So you induce the failures from outside and watch how the system degrades:

• Malformed request body. Expect a clean 400, returned quickly. A 500 or a hung connection is a defect.
• Oversized payload. Expect a 413. A truncated parse that half-succeeds is a defect.
• Exceed the rate limit. Expect a 429 with a Retry-After, and then read the other headers.
• Point a signal at an unreachable upstream. Expect a bounded timeout and a clean 5xx carrying an error you can log. Watch for a socket held open until your own client times out.

The rate-limit case is the most revealing, because it forces a design choice into the open. A distributed rate limiter has to decide what to do when its shared store is unreachable. It can fail closed, rejecting requests to protect the limit, which turns a database blip into a customer-facing auth outage. Or it can fail open, falling back to a local per-instance limit to protect availability, which loosens the global guarantee for as long as the store is down. Both are defensible. Hiding which one is in effect is not.

An honest API tells you on the wire. When our distributed check falls back to a local one, the response carries X-RateLimit-Degraded: true and an X-RateLimit-Layer header, so a caller can see that the global guarantee is temporarily not holding for that request. That is the transparency to benchmark for. Drive any vendor into its limit and read the headers: a 429 that admits nothing about its own degradation is worse than one that does, because the silent one is failing open without telling you. How verifiers should shed load without lying about it is the subject of backpressure in identity pipelines.

A benchmark you can trust

The whole method collapses to one table. For each dimension, there is a thing to measure and a shape of answer that separates a real system from a deck:

Run it yourself

The latency half of this takes about thirty seconds and no account. Open the playground, run the ten-ping burst, and read your own p50, p95, and p99 against the production edge: the latency you see is the latency you will get. The freshness and failure halves are vendor-agnostic; point the questions and the induced faults at us or at anyone else on your shortlist.

And if a vendor's answer to the boundary question is unclear, treat that as its own result. PasskeyBridge is a verification and response layer that returns a signed decision and sits alongside your identity provider rather than replacing it. A benchmark that respects that boundary is testing the right product against the right budget. One that does not will produce a number, and the number will not mean what you think it means.

Start free · Test the API