k3s AES-CBC encryption at rest feature not working

[VivekSheshadari]

Environmental Info:
K3s Version:
k3s version v1.27.4+k3s1 (36645e7)
go version go1.20.6

Node(s) CPU architecture, OS, and Version:
Linux k3d-my-cluster-server-0 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 GNU/Linux

Cluster Configuration:
1 server, 2 agents

Describe the bug:
k3s AES-CBC encryption at rest feature not working, i have tried the below steps.
started the k3s cluster with argument --secrets-encryption
and inside cluster i have tried commands below and the status show perfectly ok, but no secrets are encryted.

k3s secrets-encrypt prepare
k3s secrets-encrypt rotate

even i have tried with commands like force and reencrypt

Steps To Reproduce:
I have used k3d with below configuration to start k3s cluster in local:
Note: I have also tried the same version of k3s with argument --secrets-encryption it didn't work.

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: my-cluster
servers: 1
agents: 2
kubeAPI:
  host: "k3d.test.domain"
  hostIP: "127.0.0.1"
  hostPort: "6445"
# image: my-k3s:latest
image: rancher/k3s:v1.27.4-k3s1
# network: k3d-custom-net
# subnet: "172.28.0.0/16"
token: superSecretToken
volumes:
  - volume: C:\\Users\\Vivek.Sheshadari\\CUSTOM\\GIT\\DevOps\\K3D\\application-volume:/tmp
    nodeFilters:
      - server:0
      - agent:*
  - volume: C:\\Users\\Vivek.Sheshadari\\CUSTOM\\GIT\\DevOps\\K3D\\configs:/var/lib/rancher/k3s/server/manifests
    nodeFilters:
      - server:0
ports:
  - port: 8080:80
    nodeFilters:
      - loadbalancer
env:
  - envVar: bar=baz
    nodeFilters:
      - server:0
hostAliases:
  - ip: 127.0.0.1
    hostnames: 
      - localhost
options:
  k3d:
    wait: true
    timeout: "60s"
    disableLoadbalancer: false
    disableImageVolume: false
    disableRollback: false
    loadbalancer:
      configOverrides:
        - settings.workerConnections=2048
  k3s:
    extraArgs:
      - arg: "--cluster-init"
        nodeFilters:
          - server:*
      - arg: "--tls-san=k3d.test.domain"
        nodeFilters:
          - server:*
      - arg: "--kube-apiserver-arg=encryption-provider-config=/var/lib/rancher/k3s/server/manifests/secret_config.yaml"
        nodeFilters:
          - server:*
      - arg: "--secrets-encryption"
        nodeFilters:
          - server:* 
    nodeLabels:
      - label: role=agent
        nodeFilters:
          - agent:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  runtime:
    # gpuRequest: all
    labels:
      - label: role=agent
        nodeFilters:
          - agent:*
    ulimits:
      - name: nofile
        soft: 26677
        hard: 26677

• Installed K3s:

Expected behavior:
The secret should be encrypted as per AES_CBC standards and I should not able to decode the secret with base64 command locally.

Actual behavior:
The secret should be encrypted as per AES_CBC standards and i should not able to decode the secret with base64 command locally.

Additional context / logs:
when i run "k3s secrets-encrypt status"

Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2024-02-23T11:42:33Z
        AES-CBC   aescbckey

[brandond]

Please check the upstream docs:
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#verifying-that-data-is-encrypted

Note that encryption of API data at rest means at rest within the datastore. It is encrypted when written to the datastore, and decrypted when read, before being returned to API clients. It does not make the data inaccessible via the kubernetes API. Secrets encryption would not be much use if the data you stored in the Secret resource was inaccessible via the API.

[VivekSheshadari]

what you mean exactly when you say if data you stored in the Secret resource was inaccessible via the API ?
I have referred the same link and tested the behaviours after encryption is enabled. it is accessible via API and when I call "kubectl get secret secret1 -n default -o yaml" i could able to see the secret still in base64 and not encrypted.
Also, If you think there is any other alternate way to achieve this, provide the solution here.

[brandond]

I have referred the same link and tested the behaviours after encryption is enabled. it is accessible via API

Please read it again, perhaps more carefully. Note where it says:

any newly created or updated Secret (or other resource kinds configured in EncryptionConfiguration) should be encrypted when stored.
To check this, you can use the etcdctl command line program to retrieve the contents of your secret data.

It specifically notes that you can only view the encrypted version of the secret directly from the etcd datastore.

If you are not familiar with Kubernetes, the etcd datastore is where the data actually lives. The apiserver itself has no persistence and only provides validation of, access to, and RBAC for resources stored in the datastore. If you want to see the encrypted secret, you must bypass the apiserver and view the data directly in the datastore.

what you mean exactly when you say if data you stored in the Secret resource was inaccessible via the API ?

How would your encrypted Secret be of any use if you could not access it? Say you put a username and password in the secret, and then encrypted it. If you wanted to use the password in a pod, but it was encrypted, what good would that be to you? It would not be of any use - the secret value is only of any use if it is decrypted when read via the apiserver.

[VivekSheshadari]

I understand, but is there anyway to validate/verify if it is actually encrypting at etcd for k3s? if yes, then please provide the steps.

[brandond]

Yes, query the etcd datastore as described in the upstream documentation that I previously linked.