Engineering · 2026-09-24

Pepper Custody: Where the Secret Behind Keyed Hashing Lives

By J. W. Bouckaert

One secret under every digest

In an earlier post we moved every low-entropy identifier the platform stores, phone numbers, email addresses, IP addresses and credential subjects, from plain SHA-256 to HMAC-SHA-256 (RFC 2104) keyed on a server-held secret we call the pepper. That change is what makes a leaked digest column useless for looking people up: without the pepper, nobody can compute the digest of a candidate phone number to compare against.

It also concentrates the property in one place. The pepper is global, one value for the whole platform, because cross-tenant threat correlation depends on the same identifier producing the same digest everywhere. So the enumeration resistance of every stored digest now depends on one value, and custody of that value becomes a control in its own right.

This post covers that custody: where the pepper is kept, how it is rotated, what happens if it is lost, and what evidence exists for each claim.

Kept outside the database

The pepper is an edge-function secret in the platform's secret store. It is not a row in any table, and it is not in the database vault. We checked the vault on production while writing this: it holds five named secrets, none of them the pepper.

The separation is the point. A database dump, a leaked backup or a SQL injection yields digests and no key to test candidates against. An attacker would need to compromise the database and, separately, the runtime environment of the functions that do the hashing.

The pepper is also independent of the platform's encryption master key. Neither is derived from the other, so the loss or rotation of one leaves the other untouched.

Set, list, never read back

The secret store accepts a value and reports that a secret of that name exists. It does not return the value, through the dashboard or the management API. That is a sound property for a secret store and it has a direct consequence for rotation.

A rotation needs the outgoing value. The platform cannot provide it. Whoever rotates the pepper must already hold the current value in their own custody, outside the platform, or the rotation cannot keep old rows matchable.

Rotation, step by step

The code supports rotation without a flag day. The protocol:

Copy the current pepper into IDENTIFIER_HASH_PEPPER_PREV.
Set IDENTIFIER_HASH_PEPPER to a new value. Every new write is keyed under the new value from the next call.
Leave the previous value in place while rows written under it are still looked up by equality. Reads compute the candidate digest under both values, so an old row and a new row for the same person both match.
Once those rows have aged out or been rewritten, clear the previous value.

Three implementation details make this work in a serverless runtime:

The imported key is cached by the pepper's value, not as a singleton, so a rotation takes effect on the next call in an isolate that is already warm. The cache holds at most four keys, which covers the current and previous value with room to spare.
A previous value shorter than the minimum is ignored rather than fatal, so a mistyped entry in the previous slot cannot take reads down.
Attestations survive the overlap. When our carrier lookup forwards a digest it already keyed, it attaches an HMAC under the pepper proving so. Verification accepts an attestation made under either value while a rotation is open, so a request in flight across the switch is not refused.

Production has no previous value set today, so no rotation window is open. The protocol is written down and its code paths are covered by unit tests. Our change records show no rotation of the production pepper, so the procedure should be treated as untested in production: its first real run is where problems will appear.

Losing the outgoing value

If the current pepper is replaced and the old value is not carried into the previous slot, nothing errors. Writes succeed under the new value. Reads compute the new digest, and it matches none of the rows written before the switch.

The effect is silent. A lookup that should find an existing guardian, a prior event for the same line or an earlier record for the same subject finds nothing, and code that deduplicates on a digest inserts a second row where it should have found the first. There is no way back afterwards. The platform keeps no plaintext identifiers by design, so there is nothing to re-key, and the old value cannot be recovered from the secret store.

This is the same shape as a lost encryption key, with one difference worth stating: the data is not lost. Every row is still there. What is lost is the ability to connect a new observation of a person to their history, which for a fraud signal is most of its value.

Staging keeps its own pepper

Our staging project has a different pepper from production. We confirmed the two values differ, without reading either, before writing this.

It matters in both directions. A staging digest can never be used to test guesses against production data, and a production row copied into staging for debugging matches nothing there. A shared pepper would make every staging environment, with its weaker access controls, a place to test candidates against production digests.

Evidence an auditor can check

The claims above are checkable, and this is the evidence we point reviewers to:

Fail closed. The hashing helper throws when the pepper is missing or shorter than 16 characters, and never falls back to an unkeyed digest. A missing pepper makes every keyed write fail visibly instead of silently storing enumerable digests.
Version stamps. Stored digests carry a version saying which scheme produced them, so a reviewer can count what share of a table is keyed and what share predates keying.
A live self-test. An operator-run self-test writes a real audit row through the production write path, reads it back, and asserts that the stored IP digest is keyed, recomputable under the live pepper, and free of the plaintext address. An operator runs it when needed; nothing runs it on a schedule.
Location. The database vault's contents can be listed by name to show the pepper is absent from it.

Gaps we have not closed

Custody outside the platform is an operational control. Holding the current value somewhere the rotation can reach it is a practice, and no code enforces it.
Rotation is untested in production. No production rotation appears in our records. The first one should be rehearsed on staging with production-shaped data before it is run for real.
Presence is not checked on a schedule. Failure is loud because hashing fails closed, but the self-test that proves the live pepper keys a real row runs only when someone invokes it.

A custody checklist for your own pepper

If you key identifier digests under a secret, these questions decide whether the privacy property holds over time:

Where does the pepper live, and is it absent from every database and backup?
Can your secret store return the value? If it cannot, where is the copy a rotation will need?
Can reads match under two values at once, and is there a written step for closing that window?
Does a missing pepper stop writes, or does it fall back to something weaker?
Do your non-production environments use their own pepper?

The mechanism is covered in the keyed hashing post, and the contract language that tests a vendor's version of this is in six contract clauses that test a zero-PII claim.

Start free · Test the API