Runbooks & opérationsSecret rotation runbook

Secret rotation runbook

When and how to rotate the secrets that gate session validity, token encryption, and webhook authenticity. Keep this short — read it before the rotation, not during.

When and how to rotate the secrets that gate session validity, token encryption, and webhook authenticity. Keep this short — read it before the rotation, not during.

Inventory

SecretUsed byImpact when rotated
NEXTAUTH_SECRETJWT signing for every user session and, when no dedicated TOKEN_ENCRYPTION_KEY is set, the root the AES-256-GCM token key is derived from (currentSecret() in src/lib/security/crypto.ts takes TOKEN_ENCRYPTION_KEY, then the canonical NEXTAUTH_SECRET). Also the fallback root of INTELLIGENCE_ANON_SECRET and VITALS_BEACON_SECRET.Every active session invalidated (users log in again) and, absent a dedicated TOKEN_ENCRYPTION_KEY, every stored OAuth token stops decrypting under the current key — recoverable only by carrying the old value in TOKEN_ENCRYPTION_KEYS_LEGACY. This row said "JWT signing" and nothing else until 2026-09 : that reading is what replays backlog/_archive/security-identity/0004-crypto-key-drift-stale-tokens.md.
TOKEN_ENCRYPTION_KEYAES-256-GCM envelope for Connector.accessToken/refreshToken (Google, Meta, Notion, Shopify), IntegrationConnection.accessToken/refreshToken + metadata.clientSecretEncrypted, WhatsappChannel.{verifyToken,accessToken,appSecret}Encrypted, McpConnector.{authToken,clientId,clientSecret}Encrypted, VerifiedRevenue.encryptedToken, KycVerification.encryptedReport. There is no BYOK key store any more (retired at the AI Gateway migration) and no ProviderApiKey model — this row named both until 2026-09.Rows encrypted under the previous key are recoverable, not lost : decryptToken tries every key from getCandidateKeys(). Carry the old value in TOKEN_ENCRYPTION_KEYS_LEGACY and the encrypt-tokens cron re-encrypts them. Rotate without it and connectors must be reconnected.
TOKEN_ENCRYPTION_KEYS_LEGACYComma-separated retired encryption secrets, newest first — tried on decrypt only, never for a new write (crypto.ts:57-78)Dropping an entry before the backfill reports zero legacy rows makes those rows unrecoverable.
STRIPE_WEBHOOK_SECRETHMAC verification on Stripe webhook eventsStripe webhooks return 400 until Stripe is updated with the new endpoint secret.
WHATSAPP_APP_SECRET (+ per-channel appSecretEncrypted)HMAC on Meta-signed inbound webhooksWhatsApp inbound returns 401 until Meta re-pushes events signed with the matching secret.
CRON_SECRETBearer token gate on every /api/cron/* routeAll scheduled jobs return 401 until Vercel Cron is updated.
TURNSTILE_SECRET_KEYCloudflare challenge on auth flowsAuth pages fail closed until Cloudflare is updated.
KV_REST_API_TOKEN / UPSTASH_VECTOR_REST_TOKENUpstash authRate-limit + memory layer go read-only / down until env is updated.
RESEND_WEBHOOK_SECRETSvix signature on /api/webhooks/resendDelivery / bounce / complaint events return 401 until Resend is updated. Empty = every delivery is rejected, so nothing is recorded.
QSTASH_TOKEN + QSTASH_CURRENT_SIGNING_KEY / QSTASH_NEXT_SIGNING_KEYPublishing background jobs, and verifying the signature on incoming job callbacks (services/jobs/verifier.ts tries both keys)Enqueue fails and every job callback is rejected. Because both keys are tried, Upstash's own rollover is the safe path : the new value lands in NEXT first, then gets promoted to CURRENT.
VITALS_BEACON_SECRETNot a vitals-only secret : the per-store beacon sub-key and the HMAC of customer emails (shopify/ingester/lib/email-hash.ts) both derive from it. The readers are pinned by src/env/shared-secrets.test.ts. Falls back to NEXTAUTH_SECRET.Destructive and silent. Rotating re-keys the customer-identity hash : rows written afterwards no longer join to the same shopper, with no error, no log, no alert.
INTELLIGENCE_ANON_SECRETSalt for anonymised store identifiers, and the key of the panel pseudonym /api/intelligence/panel/ingest derives from the session user (HMAC(anonSecret, "panel:" + userId)). Falls back to NEXTAUTH_SECRET (env/server.ts), so it moves whenever that secret moves.Identifiers change. Rows anonymised before the rotation no longer match rows anonymised after it — silently, like the beacon secret above.
EVI_CLM_SECRETShared secret the Hume CLM forwards to /api/evi/chat/completions (x-evi-secret or bearer)Voice sessions return 401 until Hume is updated. Empty in production = 503, by design : the route is otherwise an unauthenticated token tap.
TRACKING_SYSTEM_PASSWORDShared password compared (timing-safe) by /api/tracking/unlock, which mints the unlock cookie requireTrackingUnlock reads (src/app/(minimal)/setup/tracking/_lib/unlock-gate.ts). Since security-identity/3004 it gates the tracking system's home (/setup/tracking) and its interactive pages (GATED_SETUP_PAGES), NOT the guides under /setup/tracking/<guide>, which are public.The unlock form rejects every password until the new value is circulated, and every existing unlock cookie stops verifying (it is an HMAC of the password). Empty = 500 on the form and a closed gate.

When to rotate

TriggerPriority
Suspected leak (env var copy-pasted in Slack/Discord, repo accident, machine theft)Within 1 hour.
Employee offboarding with access to env vars24 hours.
Annual hygiene rotationOnce per year.
New encryption requirement (e.g. SOC 2)Per the requirement.

NEXTAUTH_SECRET — full procedure (most common)

The JWT strategy means our sessions are not revocable individually. The only kill-switch is a secret rotation. Plan a maintenance window of ~5 minutes when every active user will be logged out.

Step 0 is not optional. If TOKEN_ENCRYPTION_KEY is unset, this secret IS the token-encryption key, and steps 1-5 below make every stored OAuth token undecryptable. The paragraph after step 5 says why ; this step is how you never have to read it.

  1. Check whether a dedicated key exists :
    • TOKEN_ENCRYPTION_KEY set in production → nothing to do, go to 1.
    • unset → run the TOKEN_ENCRYPTION_KEY procedure below FIRST, with the current NEXTAUTH_SECRET carried in TOKEN_ENCRYPTION_KEYS_LEGACY. It moves token encryption onto its own key and decouples the two rotations, permanently. Do it once, and this step is never needed again.
  2. Generate a new secret : openssl rand -base64 64
  3. Update Vercel env vars in this order :
    • Production : NEXTAUTH_SECRET → new value
    • Preview : same value (keeps preview previews logged in)
  4. Trigger a redeploy on main (Vercel handles the env propagation atomically).
  5. Wait for the deploy to go live (~2 min). The instant the new build serves, every old JWT fails signature check → user lands on /auth.
  6. Smoke test : open an incognito window, complete an OTP login, verify a deep link page (e.g. /[orgSlug]/~/memory) loads.

No DB migration, no code change. The JWT in old browser tabs just stops verifying on its next refresh.

But this rotation is not session-only. currentSecret() in src/lib/security/crypto.ts derives the AES-256-GCM token key from TOKEN_ENCRYPTION_KEY, then the canonical NEXTAUTH_SECRET. With no dedicated TOKEN_ENCRYPTION_KEY, step 2 above also changes the encryption key and every stored OAuth token stops decrypting under it — that is item 0004, replayed. So before step 2, do one of :

  • set a dedicated TOKEN_ENCRYPTION_KEY first (see the section below), which decouples the two for good ; or
  • add the OLD secret to TOKEN_ENCRYPTION_KEYS_LEGACY in the same env update, so decryptToken keeps recovering those rows while the encrypt-tokens cron re-encrypts them.

INTELLIGENCE_ANON_SECRET and VITALS_BEACON_SECRET inherit the same root when left unset : rotating this secret also moves anonymised handles and the customer-identity hash. Until 2026-09 this paragraph read "No DB migration. No code change." and stopped there, which is precisely the reading that breaks connectors.

TOKEN_ENCRYPTION_KEY — destructive, plan ahead

decryptToken does not return null on the first key mismatch. It tries every key from getCandidateKeys() (crypto.ts:57-78) — the current secret, then TOKEN_ENCRYPTION_KEY, NEXTAUTH_SECRET, then each comma-separated entry of TOKEN_ENCRYPTION_KEYS_LEGACY — and GCM's auth tag makes that safe : a wrong key fails final(), it never yields a plausible-but-wrong plaintext.

The staged migration this section used to prescribe is already shipped. There is nothing to patch in crypto.ts, no TOKEN_ENCRYPTION_KEY_NEXT env var exists anywhere in the repo, and the "one-shot script" is a cron. The old Path A sent operators to write code that had been merged, and the old Path B offered a connector wipe as an acceptable choice when step 1 below avoids it for free.

Procedure (the only one)

  1. Put the CURRENT secret — whatever currentSecret() resolves to today : TOKEN_ENCRYPTION_KEY if set, otherwise NEXTAUTH_SECRET — into TOKEN_ENCRYPTION_KEYS_LEGACY, comma-separated, newest first.
  2. Set TOKEN_ENCRYPTION_KEY to the new value (openssl rand -base64 64). Always rotate onto a dedicated key : it decouples token encryption from the session secret, so the next NEXTAUTH_SECRET rotation really is session-only.
  3. Redeploy. Reads keep working immediately : new writes use the new key, old rows still decrypt through the legacy candidate.
  4. Let the backfill run, or trigger it : GET /api/cron/encrypt-tokens (daily 02:30 UTC in vercel.json) classifies every Connector and IntegrationConnection row with classifyToken and enqueues an encrypt-oauth-row QStash job for each plaintext / legacy one ; reEncryptIfLegacy rewrites it under the current key. The response carries a per-status inventory.
  5. When the inventory shows zero legacy and zero unrecoverable, drop the old value from TOKEN_ENCRYPTION_KEYS_LEGACY and redeploy.

What the cron does not sweep. It covers Connector and IntegrationConnection only. WhatsappChannel, McpConnector, VerifiedRevenue.encryptedToken and KycVerification.encryptedReport are read through the same candidate-key path, so they keep working while the legacy key is present — but nothing re-encrypts them. Keep the retired secret in TOKEN_ENCRYPTION_KEYS_LEGACY for as long as those rows matter, or re-enter them by hand.

If the old secret is genuinely gone

Rows whose key no longer exists anywhere are classified unrecoverable and the cron logs cron.encrypt_tokens.unrecoverable_rows. The only remedy is a reconnect : email each org owner a reconnect link per broken connector. This is a recovery path after a mistake, not a rotation strategy.

Webhook secrets

STRIPE_WEBHOOK_SECRET and WHATSAPP_APP_SECRET rotations require a coordinated change on the third-party side :

  1. Generate new secret on Stripe / Meta.
  2. Update Vercel env with the new value.
  3. Redeploy.
  4. Within the same maintenance window, switch the third-party to send with the new secret.

Stripe specifically supports dual-signed events during rollover : keep the old endpoint webhook active alongside a new one with the new secret for ~24h, then delete the old one.

CRON_SECRET

Trivial : update the env, redeploy. Vercel Cron sends with whatever value is in env at trigger time, so no third-party coordination needed.

After any rotation

  • Log the rotation in audit-log (TODO : add automated row insertion).
  • Update the team's password manager / 1Password vault entry.
  • Note the rotation date in this file so the next hygiene rotation has a baseline.
Was this page helpful?