Skip to content
Practices7 min read

Rotating API keys without downtime: the overlap window

Issue the new key, roll it out, confirm the old one is idle, then revoke it. How AWS, Stripe and databases support overlap, and what to do when they don't.

Crunu teamPublished 7 min read

Example. Kiteline is a fictional company.
On this page

To rotate an API key without downtime, keep the old and new keys valid at the same time: issue the new key, roll it out to every service, confirm that nothing uses the old one any more, and only then revoke it. Never the other way round. That period when both keys work is the overlap window. AWS allows two active access keys per user for exactly this reason, and Stripe keeps the old key working for up to seven days after you rotate it in the dashboard. Where a provider allows only one valid credential, you build the overlap yourself, for example with two database users.

This post walks through the sequence, how the major providers support it, how to prove the old key is idle, and how to make the rollout step quick. It applies whether you rotate on a schedule, after a leak, or because someone left; the guide to what to rotate when a developer leaves covers the last case end to end.

The sequence

  1. Issue a new key at the provider while the old one still works.
  2. Store the new key where your services read it from.
  3. Roll it out: restart or reload every service that uses it.
  4. Verify: confirm the services work on the new key and that the old key gets no traffic.
  5. Revoke the old key, preferably by disabling it first and deleting it later.

Reversing steps 1 and 5 is the classic outage: revoke first, and every service using the key fails until the new one is deployed. Skipping step 4 is the quieter one: a cron job or a forgotten worker still holds the old key and breaks a day later.

Where two keys can live at once

AWS access keys. An IAM user can have two access keys, and AWS's documented procedure relies on that: "While the first access key is still active, create a second access key", "update all applications and tools to use the new access key", check whether the first key is still in use, then deactivate it rather than deleting it (AWS: update access keys). The page explains why: if something breaks, "you can switch its state back to Active to reactivate the first access key."

Shell
aws iam create-access-key --user-name deploy-bot
# roll out the new key, then:
aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE
aws iam update-access-key --user-name deploy-bot --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
# days later, once nothing has broken:
aws iam delete-access-key --user-name deploy-bot --access-key-id AKIAIOSFODNN7EXAMPLE

Stripe API keys. Stripe's dashboard has a Rotate key action with an expiration you choose. Its documentation says: "When you rotate a key in the Dashboard, both the old and new keys work for up to 7 days. This lets you migrate gradually without downtime. If you need more than 7 days, create a new key manually, migrate to it, then expire the old one when you're done" (Stripe: API keys). Choosing Now as the expiration deletes the old key immediately, which is the right choice only after a confirmed leak, when an outage is the lesser harm.

Databases. A database user usually has one password, so there is no overlap by default. AWS Secrets Manager documents two strategies. With a single user, "there is a short period of time between when the password in the database changes and when the secret is updated", during which calls may be denied. With alternating users, the rotation "clones it to create the second user" and switches between the two, so "after rotation, both user and user_clone credentials are valid" (AWS: rotation strategies). You can do the same by hand: create a second user with the same grants, move services to it, then change the first user's password and keep it as the next spare.

Everything else. Some providers let you hold several keys or tokens at once; create a new one rather than "regenerating" the existing one, since a regenerate button can revoke the old value in the same click. Check before you press it. The key rotation guides list, per service, whether old and new can overlap, for example AWS and Stripe.

Keys you sign with yourself

Some secrets aren't issued by a provider at all: session signing keys, JWT signing secrets, webhook secrets you share with partners. Here you control both sides, and the overlap is on the verifying side: sign with the new key, accept both old and new while existing sessions and tokens expire, then drop the old one.

Django builds this in. SECRET_KEY_FALLBACKS is "a list of fallback secret keys" used "to allow rotation of the SECRET_KEY." To rotate, "set a new SECRET_KEY and move the previous value to the beginning of SECRET_KEY_FALLBACKS", then remove old values "when you are ready to expire the sessions, password reset tokens, and so on, that make use of them" (Django: settings). Other frameworks and JWT libraries do the same with a list of verification keys. If yours accepts only one key, every signed session ends the moment you switch.

How to know the old key is idle

Step 4 is where confidence comes from, and these providers give you a way to check.

  • AWS returns the last time and service each access key was used through aws iam get-access-key-last-used. AWS suggests one approach is to "wait several days and then check the old access key for any use before proceeding."
  • Stripe recommends that before the old key expires you check its request logs and "expire it only after its request volume has been at zero for a few hours or days." The request logs are a menu item on each key.
  • Databases show active sessions by user; on Postgres, pg_stat_activity lists connections with their user name.

Then disable before you delete. A disabled key that turns out to be needed can come back in seconds; a deleted one can't.

How long the window should be

Long enough for every consumer to switch, and no longer.

For a routine rotation, the window is set by your slowest consumer: a weekly batch job means at least a week. For a leak, the window is a risk you're choosing to keep open, so shrink it: roll out first, revoke as soon as the services are on the new key, and accept that a forgotten consumer may break.

The window is also easy to leave open. Stripe's seven days expire on their own. AWS's second key and a spare database user stay valid until you act. Put the revoke step on a calendar or in the same ticket as the rollout.

Example. Kiteline is a fictional company.

Making the rollout step fast

Step 3 can be the slow one. If a key lives in a dozen .env files on a dozen servers, someone has to edit each one and restart each service, and the window stays open until they do.

Central storage with automatic restarts makes the rollout a single change. With Crunu, a service runs through the agent:

Shell
crunu run --env payments-api/production --restart on-change -- /opt/payments/bin/server

When someone saves a new version of the environment with the new key, crunu run notices, waits until there have been no further changes for 2 seconds, sends SIGTERM, waits up to the stop timeout, 30 seconds by default, and starts the program again with the new variables. Only the names of changed variables are logged. On a machine running the agent service, changes are reported by the service; with a person's session or a service token, crunu run checks every poll interval, 10 seconds by default. The crunu run reference has the options.

Example. Kiteline is a fictional company.

If the new key turns out to be wrong, roll back the same way. History keeps the last 100 versions of the environment, with each change listed by variable name, never by value. Restore… saves an earlier version as a new one, and the services restart on it. The old key still works during the overlap, which is exactly what makes this safe.

Two boundaries matter here. A program that caches the key in a long-lived connection pool may keep using the old one until it restarts, and a restart is what --restart on-change gives you. And Crunu doesn't issue or revoke keys at providers. Steps 1 and 5 remain in the provider's console or CLI; Crunu makes steps 2 and 3 a single save.

A checklist for each rotation

The glossary entries for the overlap window and credential reissue explain the terms, and how a reissue differs from rotating Crunu's own encryption keys.

Sources