AW Dev Rethought

✂️ Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away - Antoine de Saint-Exupéry

Security Realities: Secrets Rotation Is Easy Until You Have Consumers


Every security policy mandates rotation. Ninety days, usually. Every auditor asks for evidence of it. And in most organisations, the honest answer is that a handful of credentials get rotated diligently, a larger number got rotated once during the compliance push, and the rest have been live since before the current team arrived.

This isn’t laziness. Rotating a secret sounds like a one-line operation — generate a new value, write it to the store — and it is, right up until something other than you is reading that value. After that it stops being a security task and becomes a distributed systems problem that security happens to have asked for.


Rotation Is Not an Update

The mental model that causes the trouble is that a secret lives in one place and rotation changes it there.

A secret lives in the manager, in the memory of every running process that read it at startup, in environment variables baked into container specs, in CI configuration, in a sidecar’s cache, in a vendor’s dashboard, in a Grafana datasource somebody set up in 2023, and in at least one .env file on a laptop.

Writing a new value to the source of truth changes exactly one of those. Everything else is a copy with its own refresh behaviour, and rotation is the process of converging all of them — a distributed state transition across systems with different update mechanisms, different timings, and in some cases no update mechanism at all.


The Consumers You Cannot Enumerate

This is the part that actually stops teams.

The credential was created four years ago for one service. Since then it’s been copied into a Lambda, a scheduled job on a VM nobody logs into, a partner’s integration that was set up over email, a monitoring datasource, and a script that produces the monthly finance report and runs on one person’s machine.

There is no registry. Nobody wrote it down, because at each individual point of reuse writing it down wasn’t anyone’s job.

The only reliable way to find every consumer of a credential is to break it and see who complains.

Which is precisely why rotation gets deferred. The task isn’t “rotate the key,” it’s “rotate the key and accept responsibility for an unknown number of unrelated systems failing at an unpredictable time.” Nobody volunteers for that on a Thursday afternoon, and the quarterly reminder gets snoozed again.


Why the Failure Arrives Hours Later

When rotation does break something, it rarely breaks immediately, which makes it far harder to diagnose.

Long-lived connections don’t re-authenticate. A connection pool authenticates at startup and holds the session. Rotating the database password doesn’t disturb existing connections at all — it breaks the next one. So nothing happens, and then four hours later a pod restarts, or a connection hits its max lifetime and gets recycled, and one instance starts failing while the others are fine.

Caches have their own clocks. Secret manager clients cache with a TTL. Sidecars refresh on an interval. Each layer converges on its own schedule, so you get a window where some consumers have the new value and some have the old one.

Environment variables never refresh. This one deserves highlighting. A secret injected into a Kubernetes pod as an environment variable is fixed for the life of that pod. Mounted as a volume it will eventually update; as an env var it will not, ever, without a restart. Teams rotate the secret, see the Secret object updated, and assume propagation happened.

The combined effect is that a rotation looks successful, and then produces a scattered, partial, delayed failure that nobody connects to the change made that morning.


Two Slots Change Everything

The technical property that makes safe rotation possible is simple: the ability to have two credentials valid at once.

With two slots, rotation is a sequence with no sharp edge. Issue the new credential while the old one still works. Roll consumers over at whatever pace they can manage. Watch the usage telemetry until the old credential shows zero use. Then revoke it.

Without two slots, rotation is a cutover. There is a moment when the old value stops working and the new one starts, every consumer must cross it simultaneously, and consumers you didn’t know about cross it by failing.

A great many systems only support one. A database user has one password. Plenty of SaaS products have a single API key field with a “regenerate” button that invalidates the old one instantly.

If you build anything that issues credentials to other teams, supporting two concurrently active credentials is the single highest-leverage design decision you can make for your consumers. It is also one almost nobody thinks about at design time, because at design time there is only one consumer and they’re in the room with you.


The Real Answer Is Not Rotating at All

Step back far enough and rotation is a mitigation for a design choice.

The reason you rotate is that a long-lived credential accumulates risk with age — more copies, more exposure, more former employees who once saw it. Rotation caps that by resetting the clock periodically.

The alternative is credentials that don’t live long enough to accumulate anything. Workload identity, IAM role assumption, OIDC federation between your CI and your cloud, dynamic secrets issued per-session with a fifteen-minute lifetime. When credentials are issued automatically and expire on their own, rotation is continuous, invisible, and nobody schedules it.

Rotation is a workaround for not having identity.

Every long-lived secret you eliminate is a rotation you never have to perform, a consumer registry you never have to maintain, and a 3am incident that never happens. Teams putting effort into rotation automation are frequently automating the wrong layer — the question worth asking first is why this credential needs to exist for ninety days at all.


What to Do With the Secrets You Already Have

You have long-lived secrets today, so the migration above is a direction rather than a plan for this quarter. In the meantime:

Turn on usage auditing before you touch anything. Most secret managers and cloud providers can tell you which identities and source addresses used a credential and when. That’s how you enumerate consumers without breaking them — a week of logs beats any amount of asking around.

Make the consumer list a required field at creation. Imperfect and stale within months, but a stale list is dramatically better than no list.

Stop injecting rotatable secrets as environment variables where a volume mount or a runtime fetch would work.

And rotate more often, not less. The ninety-day rotation that terrifies everyone is terrifying because it’s rare. A weekly automated rotation is boring by the third week, and boring is the state you want a security control to reach.


50 Machine Learning Algorithms Cheatsheet Most downloaded free PDF 50 Machine Learning Algorithms Cheatsheet Blueprints of Intelligence — 50 ML Algorithms decoded Download free Downloaded 445 times

If this article helped you, you can support my work on AW Dev Rethought.


Rethought Relay:
Link copied!

Enjoyed this post?

Stay in the loop

New posts + weekly digest, straight to your inbox.

or

Create a free account

  • Save posts to your vault
  • Like posts & build history
  • New-post alerts

Comments

Add Your Comment

Comment Added!