Skip to content

2. Encrypting Secrets in etcd (Encryption at Rest)

Time to Complete

Planned time: ~30 minutes

By default, Kubernetes stores Secrets in etcd as base64-encoded plaintext. Anyone with access to etcd can read all your secrets. Encryption at rest ensures that sensitive data is encrypted before being written to etcd, providing defense-in-depth even if an attacker gains access to the etcd data files or backups.

In this lab, you’ll enable encryption at rest for Kubernetes Secrets, verify that the data is actually encrypted in etcd, and perform a key rotation workflow.

What You’ll Learn

  • How Kubernetes stores Secrets in etcd and why base64 is not encryption
  • How to configure the kube-apiserver for encryption at rest
  • How to verify that Secrets are encrypted in etcd
  • How to re-encrypt existing Secrets after enabling encryption
  • How to perform encryption key rotation
  • (Bonus) How to use different encryption providers (AES-GCM, Secretbox)
  • (Bonus) How to encrypt ConfigMaps and other resources
Trainer Instructions

Tested versions:

  • Kubernetes: 1.32.x
  • kind: 0.25.0
  • etcdctl: 3.5.x (included in etcd pod)

Cluster requirements:

  • This lab requires control plane access to modify the kube-apiserver configuration
  • Works with kubeadm-based clusters or kind (with custom configuration)
  • Not suitable for managed Kubernetes services (EKS, GKE, AKS) where you cannot modify API server flags

For kind clusters, use the provided kind-config.yaml with encryption pre-configured.

Info

We are in the cluster created by hand: kx c<x>-byhand

Prerequisites

Cluster Access

You need access to the control plane node to:

  • Place the encryption configuration file
  • Modify the kube-apiserver manifest (for kubeadm clusters)
  • Or use kind with a custom configuration

Using kind (only if the byhand cluster is broken)

If you’re using kind, copy the encryption config to the expected location and create a cluster:

Info

cp ~/exercise/kubernetes/etcd-encryption/encryption-config.yaml /tmp/encryption-config.yaml
kind create cluster --name etcd-lab --config ~/exercise/kubernetes/etcd-encryption/kind-config.yaml --image kindest/node:v1.32.0

1. Understand the Baseline: Unencrypted Secrets

Before enabling encryption, let’s understand how Kubernetes stores Secrets by default.

Create a Test Namespace and Secret

Create a namespace for this lab and a sample secret:

Solution 

kubectl create namespace etcd-lab
kubectl -n etcd-lab create secret generic demo-secret --from-literal=token=supersecret

View the Secret via kubectl

Retrieve the secret using kubectl:

kubectl -n etcd-lab get secret demo-secret -o yaml

Questions

  • Is the secret value encrypted or just encoded?
  • Can you decode it to see the original value?
Answers
  • The value is base64-encoded, not encrypted. Base64 is an encoding scheme, not encryption.
  • Yes, you can decode it easily: 
    kubectl -n etcd-lab get secret demo-secret -o jsonpath='{.data.token}' | base64 -d
    This proves that kubectl get secret -o yaml does not prove encryption at rest.

Info

Encryption at rest happens in the kube-apiserver before data is written to etcd. The API server encrypts when writing and decrypts when reading, so kubectl always shows the decrypted value.

2. Examine the etcd Storage Path

Kubernetes stores all objects in etcd under the /registry/ prefix. Understanding this path is essential for verifying encryption.

etcd Key Format

Secrets are stored in the etcd pod at the path:

/registry/secrets/<namespace>/<secret-name>

Question

What is the etcd key for a Secret named demo-secret in namespace etcd-lab?

Answer 
/registry/secrets/etcd-lab/demo-secret

View Raw Data in etcd (Optional)

If your cluster runs etcd as a pod (kubeadm/kind), you can inspect the raw stored value:

Find the etcd pod:

ETCD_POD=$(kubectl -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}')
echo "etcd pod: $ETCD_POD"

Query etcd directly:

kubectl -n kube-system exec $ETCD_POD -- sh -c '
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/etcd-lab/demo-secret
'

Without encryption, you’ll see the secret value in the output (mixed with protobuf binary data):

/registry/secrets/etcd-lab/demo-secret
k8s


v1Secret�
�

demo-secreetcd-lab"*$0a079276-63bd-4999-bc71-24b0654db3ae2Ȋ���b
kubectl-createUpdatevȊ��FieldsV1:.
,{"f:data":{".":{},"f:token":{}},"f:type":{}}B
token
        supersecretOpaque"

---

3. Enable Encryption at Rest

Now let’s configure the kube-apiserver to encrypt Secrets before storing them in etcd.

Understand the EncryptionConfiguration

Review the encryption configuration file (~/exercise/kubernetes/etcd-encryption/encryption-config.yaml):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        # Generate your own key: head -c 32 /dev/urandom | base64
        secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LWZvci1hZXM=
  - identity: {}

Info

Provider order matters! The first provider is used for encryption (writing). All providers are tried for decryption (reading). The identity provider at the end allows reading unencrypted data during migration. Read more about the full encryption configuration here

Configure the API Server (kubeadm clusters)

For kubeadm-based clusters, you need to:

  1. Place the encryption config on the control plane node
  2. Update the kube-apiserver manifest

Info

Use the kubernetes.io page to search for the instructions how to do this.

Hint

see https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#use-the-new-encryption-configuration-file

Solution

Step 1: Copy the encryption config to the control plane node (node 0):

# code vm
scp exercise/kubernetes/etcd-encryption/encryption-config.yaml azuser@NODE-0:/home/azuser/encryption-config.yaml
# now on Node 0
sudo mkdir -p /etc/kubernetes/encryption
sudo cp encryption-config.yaml /etc/kubernetes/encryption/encryption-config.yaml
sudo chmod 600 /etc/kubernetes/encryption/encryption-config.yaml

Step 2: Edit /etc/kubernetes/manifests/kube-apiserver.yaml:

Add the flag under spec.containers[0].command:

- --encryption-provider-config=/etc/kubernetes/encryption/encryption-config.yaml

Add a volume mount:

volumeMounts:
- name: encryption-config
  mountPath: /etc/kubernetes/encryption
  readOnly: true

Add a volume:

volumes:
- name: encryption-config
  hostPath:
    path: /etc/kubernetes/encryption
    type: DirectoryOrCreate

Save the file. The kubelet will automatically restart the API server.

Verify API Server is Running

After configuration changes, confirm the API server is healthy:

kubectl get nodes
kubectl -n kube-system get pods -l component=kube-apiserver

Troubleshooting

If the kube-apiserver does not respond any more, check its container logs on NODE-0 

# NODE-0
sudo crictl logs $(sudo crictl ps -aq --name kube-apiserver)

Questions

  • Which component reads the encryption configuration?
  • Why do we keep identity as a fallback provider?
Answers
  • The kube-apiserver reads the encryption config and performs encryption/decryption.
  • identity allows the API server to read unencrypted data that existed before encryption was enabled. You can remove it after re-encrypting all existing data.

4. Verify Encryption is Working

With encryption enabled, new Secrets should be encrypted in etcd.

Create a New Secret

Create a secret after encryption is enabled in the etcd-lab namespace:

Hint 
kubectl -n etcd-lab create secret generic ...
Solution 

kubectl -n etcd-lab create secret generic post-encryption-secret --from-literal=token=encrypted-value

Verify in etcd

Check that the new secret is stored encrypted:

Hint

Access etcd like you did before, but this time print the newly created secret after encrpytion was enabled

Solution 

ETCD_POD=$(kubectl -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}')

kubectl -n kube-system exec $ETCD_POD -- sh -c '
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/etcd-lab/post-encryption-secret
'
Look for the encryption marker `k8s:enc:aescbc:v1:key1:` at the beginning of the stored value. This indicates the secret is encrypted with AES-CBC using key1.

Tip

The encryption marker format is: k8s:enc:<provider>:<version>:<key-name>:

5. Re-encrypt Existing Secrets

Enabling encryption only affects new writes. Existing secrets remain unencrypted until they are rewritten.

Why Re-encryption is Necessary

Question

Why aren’t existing secrets automatically encrypted when you enable encryption at rest?

Answer

Kubernetes does not automatically rewrite all stored objects when you change the encryption configuration. Encryption happens at write time, so existing data must be explicitly rewritten to be encrypted.

Re-encrypt All Secrets in a Namespace

Force a rewrite of all secrets to encrypt them:

Hint

Print all secrets as json and see if kubectl allows a replace in-place

Solution 

kubectl -n etcd-lab get secrets -o json | kubectl replace -f -
Or for all namespaces: 
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verify the Original Secret is Now Encrypted

Check the demo-secret that existed before encryption was enabled:

Solution 

kubectl -n kube-system exec $ETCD_POD -- sh -c '
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/etcd-lab/demo-secret
'
You should now see the `k8s:enc:aescbc:` prefix indicating encryption.

6. Bonus: Key Rotation

Bonus Exercise

This section is optional and covers encryption key rotation.

Encryption keys should be rotated periodically. The process involves:

  1. Add a new key as the first key (becomes the encryption key)
  2. Keep the old key as the second key (for decryption of existing data)
  3. Restart the API server
  4. Re-encrypt all secrets with the new key
  5. (Optional) Remove the old key after all data is re-encrypted

Info

Also search the docs at kubernetes.io

Task

  1. Update the encryption config to add a new key
  2. Restart the API server
  3. Re-encrypt all secrets
  4. Verify secrets are encrypted with the new key
Hint

Read the docs here. The rotated configuration should look like encryption-config-rotated.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key2
        # New primary key for encryption
        secret: bmV3LWtleS1mb3Itcm90YXRpb24tMzItYnl0ZXM=
      - name: key1
        # Old key kept for decryption of existing data
        secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LWZvci1hZXM=
  - identity: {}
Solution

Step 1: Update the encryption config with the new key at the top:

sudo cp encryption-config-rotated.yaml /etc/kubernetes/encryption/encryption-config.yaml

Step 2: For kubeadm clusters, touch the manifest to trigger a restart:

sudo touch /etc/kubernetes/manifests/kube-apiserver.yaml
# and verify that it restarted
sudo crictl ps -a -name kube-apiserver

Wait for the API server to restart:

kubectl -n kube-system get pods -l component=kube-apiserver -w

Step 3: Re-encrypt all secrets:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Step 4: Verify the new key is used:

kubectl -n kube-system exec $ETCD_POD -- sh -c '
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/etcd-lab/demo-secret
'

Look for k8s:enc:aescbc:v1:key2: indicating the new key is in use.

Questions

  • Why do we keep the old key during rotation?
  • When is it safe to remove the old key?
Answers
  • The old key is needed to decrypt data that was encrypted with it. Without it, existing secrets would become unreadable.
  • It’s safe to remove the old key only after all secrets have been re-encrypted with the new key.

7. Bonus: Alternative Encryption Providers

Bonus Exercise

This section explores different encryption providers.

Kubernetes supports multiple encryption providers:

Provider Description Use Case
aescbc AES-CBC with PKCS#7 padding General purpose, widely used
aesgcm AES-GCM (authenticated encryption) Better security, must rotate keys frequently
secretbox XSalsa20 + Poly1305 Modern, fast, recommended
identity No encryption (plaintext) Fallback for migration
kms External KMS provider Production (AWS KMS, GCP KMS, etc.)

Task

  1. Review the Secretbox configuration
  2. Understand when to use each provider
Secretbox Configuration

Secretbox is recommended for new deployments:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  - configmaps
  providers:
  - secretbox:
      keys:
      - name: key1
        # Secretbox requires exactly 32 bytes
        # Generate: head -c 32 /dev/urandom | base64
        secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LWZvci1hZXM=
  - identity: {}

Questions

  • Why might you choose secretbox over aescbc?
  • Why is aesgcm key rotation critical?
Answers
  • secretbox uses modern cryptographic primitives (XSalsa20 + Poly1305) and is faster than AES-CBC. It also provides authenticated encryption.
  • aesgcm must not encrypt more than 2^32 writes with the same key due to nonce collision risks. Frequent key rotation is mandatory.

Tip

For production environments, consider using a KMS provider (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault) which provides:

  • Centralized key management
  • Automatic key rotation
  • Audit logging
  • Hardware security modules (HSMs)

8. Clean Up

Remove the resources created during this lab:

kubectl delete namespace etcd-lab
Optional: Remove Encryption (kubeadm clusters)

To disable encryption and revert to plaintext storage:

  1. Update encryption config to put identity first
  2. Restart API server
  3. Re-encrypt all secrets (they’ll be stored as plaintext)
  4. Remove the --encryption-provider-config flag
  5. Restart API server again
Delete kind cluster

If using kind:

kind delete cluster --name etcd-lab

Recap

You have:

  • Understood how Kubernetes stores Secrets in etcd (base64-encoded, not encrypted by default)
  • Configured the kube-apiserver for encryption at rest using AES-CBC
  • Verified that Secrets are encrypted in etcd by examining the raw stored data
  • Re-encrypted existing Secrets to apply encryption retroactively
  • (Bonus) Performed encryption key rotation
  • (Bonus) Explored alternative encryption providers (AES-GCM, Secretbox)

Wrap-Up Questions

Discussion

  • What threats does encryption at rest protect against?
  • What threats does it NOT protect against?
  • How would you handle encryption in a managed Kubernetes service?
Discussion Points
  • Protects against: Unauthorized access to etcd data files, etcd backups, and physical disk access.
  • Does NOT protect against: API server compromise, RBAC misconfiguration, or network interception (use TLS for that).
  • Managed services: Most managed Kubernetes services (EKS, GKE, AKS) encrypt etcd by default using their cloud KMS. You typically cannot configure custom encryption providers, but you can use Kubernetes Secrets Store CSI Driver to fetch secrets from external vaults.

Further Reading

End of Lab