Secrets management

An online travel agency runs on secrets: passwords for its booking database, API keys for 40 airlines and a payment provider, the key that signs customers' login tokens, TLS certificates, credentials for a partner's file server. Such secrets usually end up hardcoded in source code in plain text, scattered through configuration files and configuration tools, and often shared by several services, so nobody can tell which one leaked.

Leaks are common and expensive. In 2016 intruders reached Uber's cloud storage using an access key that an engineer had posted on a code-sharing website, and downloaded files containing more than 25 million names and email addresses and 22 million names and phone numbers. One key, one repository, millions of people.

In 2024, 23.8 million new secrets were found in public commits across 69.6 million public repositories; private repositories, wrongly assumed safe, were worse; 70% of the secrets leaked in 2022 still worked. Each figure counts a different population, so compare them loosely.

The last bar is the real lesson. Leaks happen; what turns a leak into a breach is a secret that still works. Secrets management is as much about making leaked secrets useless quickly as about preventing leaks.

The first step is a secrets manager: one service that stores secrets encrypted, decides who may read each one, and logs every read. HashiCorp Vault, AWS Secrets Manager, Google Secret Manager and Azure Key Vault are common choices. Such a system needs a few properties:

Services fetch secrets at startup or when needed, keep them in memory, and never write them to disk, logs or images. For containers: never bake a secret into an image with Docker's ENV or ARG instructions, which can leak with the image's definition; inject secrets at runtime instead, ideally by fetching them directly from the store.

A service that fetches its secrets must first prove who it is to the secrets manager. If that proof is itself a password in a config file, the problem has only moved: the first secret is now the one to steal.

Cloud platforms solve it by giving workloads an identity they never have to store. A virtual machine has an instance profile; a Kubernetes pod can be bound to a cloud role. With Amazon EKS Pod Identity, for example, a service account is associated with an IAM role, and an agent on each node hands the pod temporary credentials, instead of credentials being created and distributed to the containers:

Even better is not needing a secret at all. If the database accepts the platform's identity directly, or services authenticate to each other with workload certificates (Network segmentation & zero trust), there is no password to leak in the first place.

A database password shared by every instance of every service for years is the worst kind of secret: widely copied, rarely changed, and impossible to trace. Dynamic secrets turn that around. Vault's database secrets engine, for example, creates a new database user with its own password when a service asks, and attaches a lease with a time to live:

Three things change. A stolen credential works only until its lease ends, and after a reboot, dynamically created credentials have simply expired. Every instance has unique credentials, so a strange query in the database log names the instance that sent it. And revoking access means revoking leases, not hunting down every copy of a shared password. Use dynamic secrets wherever possible.

Many secrets can't be created per use: an airline's API key, the payment provider's key, the token-signing key. They should be rotated regularly, so a stolen copy works only for a while, and should have an expiration date that forces rotation if nobody does it. One exception: people's own passwords should not be rotated on a schedule, only when there is evidence of compromise.

The hard part is rotating without breaking running services, which may still hold the old secret. AWS Secrets Manager offers two strategies for database credentials. A single user whose password is changed is simplest, but a request made during the change can briefly fail. Alternating users keeps two users and updates the one not currently in use:

Alternating users suit applications that need high availability: a service fetching the secret mid-rotation always gets a valid pair. The same pattern works for keys: an API that accepts both the old and the new key during a changeover, or a token signer that publishes both public keys until old tokens expire (Authentication & authorization).

Detection belongs as early as possible. Scan for secrets at the developer's machine, in the editor or a pre-commit hook, before anything is committed. Code hosts add a second net: GitHub's push protection blocks pushes that contain recognized secrets before they reach the repository, rather than alerting afterwards, and secret scanning alerts on secrets already in the history.

Some leaks are built into tools. Kubernetes Secrets are, by default, stored unencrypted in the cluster's data store, readable by anyone with API access to them, and by anyone allowed to create a pod in that namespace, since the pod can mount any Secret there. In Kubernetes, enable encryption at rest for Secrets and restrict access with role-based access control.

When a secret does leak, the order is clear, and revocation comes first:

• Revoke the exposed secret immediately, so it stops working.
• Rotate: create and deploy a replacement, ideally automatically.
• Investigate the audit logs: what was the secret used for, and by whom, since it leaked.
• Remove it from the code and history, knowing that copies may already exist elsewhere.

Deleting the commit alone achieves little: a pushed secret must be treated as public from the moment of the push.

For the travel agency, the checklist:

• One secrets manager, with least-privilege policies per service and audit logs of every read.
• No secrets in code, images, logs or chat; inject at runtime from the store.
• Platform identity for the first secret, never a password in a config file.
• Dynamic, per-instance credentials for databases where possible.
• Automated rotation without downtime for the rest, with expiry dates as a backstop.
• Scan before commit and block on push.
• Encrypt Kubernetes Secrets at rest and restrict who can read them.
• A rehearsed leak runbook: revoke, rotate, investigate, then clean up.