Your Kubernetes Secrets Are Sitting in Plaintext and That’s Fine (Until It Isn’t)
You got GitOps working. Flux is humming, ArgoCD is happy, every manifest lives in Git — except for one awkward pile of base64 database passwords that you’re definitely not committing and definitely also have no idea how to manage. Sound familiar?
Secret management in Kubernetes is one of those problems where the community solved it twice, in completely different directions, and both solutions are correct depending on who you ask. Bitnami Sealed Secrets says “encrypt the secret and commit it.” External Secrets Operator (ESO) says “keep the secret out of Git entirely and pull it from something smarter.”
Neither one is wrong. But picking the wrong one for your situation is going to cause you pain at 2 AM when you’re rotating a certificate or recovering from a node failure. Let’s sort it out.
Sealed Secrets: Git Is the Source of Truth (Encrypted)
The pitch is simple: you take your Kubernetes Secret, encrypt it with the cluster’s public key using the kubeseal CLI, get back a SealedSecret CRD, and commit that to Git. The Sealed Secrets controller running in your cluster decrypts it back into a real Secret at apply time. Nobody else can decrypt it without the cluster’s private key.
Installing It
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secretshelm repo updatehelm install sealed-secrets sealed-secrets/sealed-secrets \ --namespace kube-system \ --version 2.16.1Grab the kubeseal CLI to match:
# Linux amd64curl -sL https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.26.3/kubeseal-0.26.3-linux-amd64.tar.gz \ | tar xz kubesealsudo mv kubeseal /usr/local/bin/Sealing a Secret
Start with a regular Kubernetes Secret manifest (never committed raw):
apiVersion: v1kind: Secretmetadata: name: db-creds namespace: myapptype: OpaquestringData: DB_PASSWORD: "hunter2" DB_USER: "appuser"Seal it:
kubeseal --format yaml < db-creds-raw.yaml > db-creds-sealed.yamlOr seal a single value inline (useful in scripts):
kubeseal --raw \ --from-file=token=./github-token.txt \ --name=github-token \ --namespace=ci \ --scope=strictThe output db-creds-sealed.yaml looks like this:
apiVersion: bitnami.com/v1alpha1kind: SealedSecretmetadata: name: db-creds namespace: myappspec: encryptedData: DB_PASSWORD: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq... DB_USER: AgCH1YLtVAMtAmR7PIkiKS6y2fjIj3LN... template: metadata: name: db-creds namespace: myapp type: OpaqueCommit this. Push it. GitOps picks it up, the controller decrypts it, a real Secret appears. Done.
The One Thing You Must Back Up
The cluster has a sealing keypair stored in kube-system. If you lose it, every SealedSecret in your repo becomes an encrypted brick. Back it up:
kubectl get secret \ -n kube-system \ -l sealedsecrets.bitnami.com/sealed-secrets-key \ -o yaml > sealed-secrets-key-backup.yamlStore that somewhere safe — a password manager, an encrypted volume, not Git. Ironic, yes.
External Secrets Operator: Git Holds Nothing, a Backend Holds Everything
ESO flips the model. Your secrets live in an external system — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Bitwarden, Doppler, 1Password — and ESO pulls them into Kubernetes Secrets on a schedule. Your Git repo contains ExternalSecret CRDs that describe where to fetch the secret, not the secret itself.
Installing It
helm repo add external-secrets https://charts.external-secrets.iohelm repo updatehelm install external-secrets external-secrets/external-secrets \ --namespace external-secrets \ --create-namespace \ --version 0.10.4Wiring Up a SecretStore (Vault Example)
First, tell ESO how to talk to your backend. A ClusterSecretStore works across all namespaces:
apiVersion: external-secrets.io/v1beta1kind: ClusterSecretStoremetadata: name: vault-backendspec: provider: vault: server: "http://vault.vault.svc.cluster.local:8200" path: "secret" version: "v2" auth: kubernetes: mountPath: "kubernetes" role: "external-secrets" serviceAccountRef: name: "external-secrets" namespace: "external-secrets"Then the ExternalSecret that does the actual fetch:
apiVersion: external-secrets.io/v1beta1kind: ExternalSecretmetadata: name: db-creds namespace: myappspec: refreshInterval: 1h secretStoreRef: name: vault-backend kind: ClusterSecretStore target: name: db-creds creationPolicy: Owner data: - secretKey: DB_PASSWORD remoteRef: key: secret/data/myapp/db property: password - secretKey: DB_USER remoteRef: key: secret/data/myapp/db property: usernameESO creates the Secret and re-syncs every hour. Rotate the value in Vault, wait up to an hour (or force a sync), done — no code change, no commit, no redeployment required.
The Bootstrap Problem
Honest caveat here: ESO needs something to authenticate against. With Vault that means Vault is already running and you’ve configured the Kubernetes auth method:
vault auth enable kubernetes
vault write auth/kubernetes/config \ kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
vault policy write external-secrets - <<EOFpath "secret/data/*" { capabilities = ["read"]}EOF
vault write auth/kubernetes/role/external-secrets \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets \ policies=external-secrets \ ttl=1hThat’s real work. If you don’t already run Vault, you’re not just installing ESO — you’re also installing Vault, unsealing it, configuring it, and deciding where its unseal keys live. The turtles go all the way down.
Head-to-Head: Where They Actually Differ
Storage Location
Sealed Secrets: The encrypted blob lives in Git. Git is your source of truth. This is the whole appeal for GitOps purists — everything, including secrets, is in the repo.
ESO: Git has zero secret data, not even encrypted. The external backend is the source of truth. Your Git repo is just configuration describing what to fetch.
Secret Rotation
Sealed Secrets: You have to re-encrypt and re-commit. That means re-running kubeseal, updating the file, making a commit, and pushing. Automatable, but manual by default.
ESO: Rotate at the source (Vault, AWS SM, wherever). ESO picks it up on the next refreshInterval. If you need it now, annotate the ExternalSecret to force a sync:
kubectl annotate externalsecret db-creds \ -n myapp \ force-sync=$(date +%s) \ --overwriteMulti-Cluster
Sealed Secrets: Each cluster has its own keypair. You can share a keypair across clusters by importing it, but that’s extra work and slightly defeats the per-cluster isolation story. Managing N clusters means managing N backup keys (or sharing one very important one).
ESO: One Vault instance, all clusters authenticate to it. ClusterSecretStore references the same backend everywhere. This is where ESO genuinely shines — it scales horizontally without any per-cluster key ceremony.
Disaster Recovery
Sealed Secrets: Lose the sealing key = lose access to all sealed secrets. Recovery means restoring the key backup and reapplying. If you have the backup, it’s fine. If you don’t, you’re re-creating everything from scratch.
ESO: Lose the cluster = the secrets still exist in Vault. Rebuild the cluster, reinstall ESO, reapply your ExternalSecret CRDs from Git, and ESO re-fetches everything. Genuinely easier recovery story, assuming Vault is healthy.
Audit Trail
Sealed Secrets: Your audit trail is git log. You can see who committed what SealedSecret when, but you can’t see who read it or whether it was rotated.
ESO: The backend has its own audit log. Vault’s audit device will show you every read, every write, every token used. That’s enterprise-grade auditability, and it’s one of the main reasons security teams push for ESO + Vault in regulated environments.
The Pitfalls No One Puts in the README
Sealed Secrets: Key Rotation Is Spicy
The controller generates a new keypair every 30 days by default (configurable). Old secrets remain decryptable because old keys are kept. But if you ever run kubeseal --rotate-keys to force a rekey, you need to re-seal every SealedSecret in your repo with the new key. Miss one and it’ll fail to decrypt after the old key is pruned.
Check what keys you have:
kubectl get secret \ -n kube-system \ -l sealedsecrets.bitnami.com/sealed-secrets-keyAnd be careful: re-sealing changes the encrypted blob, which means a Git commit for every secret. Not the end of the world, but definitely not something you want to discover mid-incident.
ESO: The Vault Dependency Is Real
If Vault goes down, ESO can’t sync. Existing Secrets in the cluster keep working (they’re already there), but any new pod that needs a Secret that hasn’t been created yet will fail. ESO caches the last known value, so brief outages are usually fine. Extended Vault downtime in a recovery scenario where you’re also rebuilding your cluster is genuinely stressful.
ESO: Auth Bootstrap Is a Chicken-and-Egg
To authenticate to Vault using Kubernetes ServiceAccount tokens, Vault needs to trust your cluster’s API server. That means Vault needs to be configured before ESO is installed. In a fresh cluster rebuild, the order of operations matters and it’s easy to get wrong.
The Hybrid Pattern (Actually Pretty Clever)
Here’s something that comes up in production: use ESO for everything except the Vault unseal keys. Use Sealed Secrets for just those.
Vault unseal keys can’t live in Vault (obviously). They can’t live in plain Git. But you can seal them with Sealed Secrets, commit the SealedSecret, and have the cluster unseal Vault on startup automatically. ESO then takes over for everything else.
This solves the bootstrapping problem elegantly: Sealed Secrets handles the one secret that can’t go anywhere else, ESO handles the rest.
# vault-unseal-sealed.yaml — committed to GitapiVersion: bitnami.com/v1alpha1kind: SealedSecretmetadata: name: vault-unseal-keys namespace: vaultspec: encryptedData: unseal_key_1: AgBy3i4OJSWK... unseal_key_2: AgCH1YLtVAMt... unseal_key_3: AgDJ2ZMuWBNu... template: metadata: name: vault-unseal-keys namespace: vaultA Vault init job reads from that Secret, unseals Vault, and then ESO takes over. Clean.
Home Lab Recommendation
Start with Sealed Secrets if:
- You’re running a single cluster (k3s on a NUC, Talos on three nodes, whatever)
- You don’t already have Vault or another secret backend
- You want full GitOps without any external dependencies
- You’re the only person managing secrets
The operational overhead is low. One controller, one backup, done. The kubeseal workflow is annoying enough that you’ll be careful with secrets (arguably a feature).
Go with ESO if:
- You already run Vault or use a cloud secret manager
- You have multiple clusters and don’t want to manage per-cluster keypairs
- You need real secret rotation without commits
- Your threat model includes “someone gets read access to my Git repo”
- You’re building toward a team setup where audit logs actually matter
The hybrid is worth considering if:
- You’re going full ESO + Vault but need somewhere for the unseal keys to live
- You want Sealed Secrets as a safety net for truly bootstrap-critical secrets
Should You Bother?
If you’re running kubectl create secret and crossing your fingers that nobody looks at your Git history — yes, bother. Both of these are better than plaintext.
Sealed Secrets is the lower-friction option and it’s genuinely good enough for most home lab setups. Install it in an afternoon, seal your secrets, commit, move on. The only real failure mode is losing the keypair backup, and that’s one kubectl get secret command away from being solved.
ESO is the right call once you outgrow a single cluster or once you already have a secret backend worth integrating. It’s more moving parts upfront but a much cleaner operational story over time — especially when a secret rotation happens at 2 AM and you’d rather update one value in Vault than make a commit, push, and wait for ArgoCD to sync.
Pick the one that matches where you are today, not where you think you’ll be in two years. You can migrate between them. The sealed keypair backup you didn’t take? That one’s harder to recover from.
