External Secrets Operator + Vault on k3s

Sealed Secrets Was Fine Until It Wasn’t

You’ve been shipping Sealed Secrets in your k3s cluster for a year. It works. Developers re-encrypt secrets when they rotate. Your backups are safe because the encryption key lives in the cluster. Life is… okay.

Then your ops team asks: “What if we need to share a secret with our staging cluster without deploying new sealed-secrets keys?” Or: “Can we audit who accessed what secret and when?” Or worse: “Our security audit says we need centralized secret rotation.”

Sealed Secrets will look at you and shrug. It’s a simple lock-the-secret-in-git solution, nothing more. For a home lab, that’s enough. For anything touching production, or anything where compliance people show up with checklists, you need something that talks to a proper secret backend.

Enter External Secrets Operator (ESO) + Vault. It’s the car you take to work instead of the forklift you’ve been driving.

ESO doesn’t care where secrets live. You can use Vault, AWS Secrets Manager, HashiCorp Cloud Platform, Azure Key Vault, or even a plain HTTP API. Vault is the obvious choice for a self-hosted setup because you can run it on the same cluster, or on a separate rig if you’re paranoid about keeping secrets away from app pods.

Here’s the k3s setup: running Vault, installing ESO, and pulling secrets from Vault into your cluster without ever checking them into git.

What We’re Building

• Vault runs in a pod on your cluster (or separate, doesn’t matter)
• External Secrets Operator watches for ExternalSecret resources
• You write YAML that says “pull my-app-db-password from Vault into a Kubernetes Secret”
• ESO talks to Vault, fetches the secret, creates a Kubernetes Secret, boom, your app pod mounts it
• Vault audits every read; you can rotate passwords without redeploying
• Sealed Secrets stays in your cluster, but you’re not using it for new workloads

If you’ve got a home lab k3s cluster running and you’re tired of managing secrets manually, this is your move.

Install Vault (Standalone, In-Cluster)

First, we’ll run Vault as a stateful pod. This is not production-grade (production Vault runs in HA with a proper storage backend), but for a home lab, it’s fine. If you want HA, swap the storage backend to Consul or PostgreSQL. Vault’s documentation handles that.

Vault Namespace & Storage

Create a namespace and persistent volume for Vault’s data:

apiVersion: v1
kind: Namespace
metadata:
  name: vault

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vault-data
  namespace: vault
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-path

Apply that. k3s gives you the local-path storage class by default.

Vault Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.16
        ports:
        - containerPort: 8200
          name: http
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "dev-root-token"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        - name: vault-config
          mountPath: /vault/config
      volumes:
      - name: vault-data
        persistentVolumeClaim:
          claimName: vault-data
      - name: vault-config
        configMap:
          name: vault-config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: vault
data:
  vault.hcl: |
    ui = true
    listener "tcp" {
      address       = "0.0.0.0:8200"
      tls_disable   = 1
    }
    storage "file" {
      path = "/vault/data"
    }

---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  ports:
  - port: 8200
    targetPort: 8200
  selector:
    app: vault

Deploy it:

kubectl apply -f vault.yaml
kubectl -n vault logs -f deployment/vault

Wait for the pod to be ready. You’ll see something like “Vault server started!” in the logs.

Unseal & Configure Vault

Vault starts in “sealed” mode. You have to unseal it. In dev mode (above), there’s only one unseal key, and we’re providing a root token, so it’s… less secure but simpler.

Port-forward to Vault:

kubectl -n vault port-forward service/vault 8200:8200

Then authenticate with the CLI:

export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=dev-root-token
vault status

You should see “Sealed: false” and “Version: 1.16.x”.

Enable KV Secret Engine

Vault ships with the secret/ path disabled. Enable it:

vault secrets enable -version=2 -path=secret kv

Create a test secret:

vault kv put secret/my-app/db-password username=appuser password="super-secret-db-pass"
vault kv get secret/my-app/db-password

Good. Vault is running and storing secrets.

Install External Secrets Operator

ESO is a Kubernetes operator that watches for ExternalSecret resources and syncs them to Kubernetes Secrets.

Add the Helm Repository

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Install ESO

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

Verify it’s running:

kubectl -n external-secrets-system get pods

You should see external-secrets-webhook-* and external-secrets-* pods.

Create a Vault Auth Token for ESO

ESO needs credentials to talk to Vault. We’ll create a Kubernetes ServiceAccount and enable Vault’s Kubernetes auth method so that the ServiceAccount can authenticate to Vault.

Enable Kubernetes Auth in Vault

vault auth enable kubernetes

Configure it to talk to your cluster’s Kubernetes API:

vault write auth/kubernetes/config \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

If you’re running the Vault CLI outside the cluster, get these values manually:

kubectl -n vault exec -it deployment/vault -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

kubectl -n vault exec -it deployment/vault -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Paste them into the write command above.

Create a Vault Policy for ESO

Create a policy that allows ESO to read secrets from the secret/ path:

vault policy write eso-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read", "list"]
}
EOF

Create a Kubernetes Auth Role

vault write auth/kubernetes/role/external-secrets-role \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=default,kube-system \
  policies=eso-policy \
  ttl=1h

Replace default,kube-system with the namespaces where your apps will run.

Create a ClusterSecretStore

A ClusterSecretStore tells ESO how to authenticate to Vault. It’s a cluster-wide resource that external secrets reference.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets-role"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets-system

The serviceAccountRef points to the ESO ServiceAccount (created by the Helm chart). Vault will use the Kubernetes auth method to verify that requests from this ServiceAccount are legit.

Apply it:

kubectl apply -f cluster-secret-store.yaml

Create an ExternalSecret

Now the fun part. Write an ExternalSecret resource that pulls a secret from Vault and creates a Kubernetes Secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: app-db-secret
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        username: "{{ .username }}"
        password: "{{ .password }}"
        database_url: "postgresql://{{ .username }}:{{ .password }}@postgres.default.svc.cluster.local:5432/mydb"
  data:
  - secretKey: username
    remoteRef:
      key: my-app/db-password
      property: username
  - secretKey: password
    remoteRef:
      key: my-app/db-password
      property: password

This says:

• Fetch the secret at secret/my-app/db-password from Vault (the my-app/db-password key)
• Extract the username and password properties
• Create a Kubernetes Secret called app-db-secret with those values
• Optionally, use the template section to build derived fields (like a full DB connection string)

Apply it:

kubectl apply -f external-secret.yaml

Check that the Kubernetes Secret was created:

kubectl get secret app-db-secret -o yaml

You should see:

data:
  username: YXBwdXNlcg==  # base64-encoded "appuser"
  password: c3VwZXItc2VjcmV0LWRiLXBhc3M=  # base64-encoded password
  database_url: cG9zdGdyZXM6Ly9hcHB1c2VyOi4uLg==

If the Secret doesn’t exist, check the ExternalSecret status:

kubectl describe externalsecret app-db-secret

Look for errors like “failed to authenticate” or “key not found”. Common issues:

• Vault is sealed (run vault status and check)
• The Kubernetes auth role is misconfigured
• The secret path doesn’t exist in Vault

Using the Secret in a Pod

In your Deployment or Pod spec, mount the Secret as an environment variable or volume:

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: password
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: app-db-secret
          key: database_url

Your app reads these environment variables. No secrets in the Pod spec, no secrets in git.

Rotate Secrets Without Redeploying

Update the secret in Vault:

vault kv put secret/my-app/db-password username=appuser password="new-shiny-password"

ESO checks for updates every hour (controlled by refreshInterval: 1h in the ExternalSecret). When it detects a change, it updates the Kubernetes Secret. Your app can either restart automatically (if you use a deployment with a Prometheus sidecar that watches the Secret) or pick up the new secret on its next read.

Some apps re-read secrets on every request (smart). Others cache them in memory (less smart, but faster). The tradeoff is yours.

Vault Outside the Cluster (Optional)

If you want Vault on a separate machine or in a different cluster, just change the server: URL in your ClusterSecretStore:

vault:
  server: "https://vault.mycompany.com:8200"

And use a different auth method if Kubernetes auth doesn’t make sense. Vault supports AppRole (agent credentials), JWT (OIDC), or plain token auth.

When to Use This vs Sealed Secrets

Use Sealed Secrets if:

• Your cluster is truly immutable and you trust its encryption key
• You want everything encrypted in git and never touch an external system
• Your team is small and rotation is rare

Use ESO + Vault if:

• You need centralized secret management across clusters
• Compliance or security audits require secret rotation and audit trails
• You want to share secrets between staging and production safely
• You’re running multiple environments and hand-managing secrets is getting tedious

For a home lab with one cluster, Sealed Secrets is probably still fine. But once you’re managing anything more complex, or once you want to stop checking secrets into git even in encrypted form, ESO + Vault is the obvious upgrade. It’s not rocket science, it’s just infrastructure doing what it should: keeping secrets secret and auditable.

Troubleshooting Checklist

ExternalSecret stuck in “pending”:

• Check kubectl describe externalsecret <name> for the error
• Verify Vault is reachable: kubectl -n vault logs deployment/vault
• Verify the secret exists in Vault: vault kv get secret/my-app/db-password
• Check RBAC: is the ESO ServiceAccount allowed to query Kubernetes auth?

Vault sealed unexpectedly:

• Run vault status to check seal status
• Unseal with the unseal keys (stored somewhere safe, right?)
• For dev mode, restart the pod, it auto-unseals with the root token

“Authentication failed” in ExternalSecret status:

• Verify the Kubernetes auth role exists: vault list auth/kubernetes/role
• Check that the bound_service_account_names includes your ESO ServiceAccount
• Verify TLS (if Vault runs with TLS, make sure your ClusterSecretStore uses the correct CA cert)

Secret not updating when you change it in Vault:

• refreshInterval controls how often ESO checks, default is 1 hour
• Manually trigger a sync: kubectl annotate externalsecret <name> force-sync=$(date +%s) --overwrite
• Restart the ESO controller: kubectl rollout restart -n external-secrets-system deployment/external-secrets

Secrets out of git, auditable at every access, rotatable without redeployment. That’s the promise. ESO + Vault delivers.

Related Reading

• ArgoCD for Home Lab GitOps
• Capsule: Multi-Tenancy for Home Lab k3s
• Cilium on k3s: When eBPF Networking Pays
• cri-o vs containerd
• Crossplane vs Terraform for Home Lab