dex secret not found

[donydonald1]

hello @khuedoan
i am a big fan of this and I have been trying to get work like this a week now but I have a little issue trying to make mine work. hoping you could help.
external secret didn't create a secret for dex.
please help

)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dex
    Optional:    false
  kube-api-access-wl57w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason  Age                    From     Message
  ----     ------  ----                   ----     -------
  Warning  Failed  60m (x12 over 62m)     kubelet  Error: secret "dex-secrets" not found
  Normal   Pulled  2m31s (x278 over 62m)  kubelet  Container image "ghcr.io/dexidp/dex:v2.38.0" already present on machine ``` 

[khuedoan]

Hi, dex-secrets is created by https://github.com/khuedoan/homelab/blob/master/platform/dex/templates/secret.yaml, could you please post the output of:

kubectl describe -n dex externalsecret dex-secrets

[donydonald1]

+ kubectl describe -n dex externalsecret dex-secrets
Name:         dex-secrets
Namespace:    dex
Labels:       argocd.argoproj.io/instance=dex
Annotations:  <none>
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  Creation Timestamp:  2024-04-18T09:48:36Z
  Generation:          1
  Resource Version:    51172
  UID:                 94eb9cd1-310b-4a3d-8574-7ed4b326de5c
Spec:
  Data:
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_id
    Secret Key:             KANIDM_CLIENT_ID
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             KANIDM_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.grafana
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GRAFANA_SSO_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.gitea
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GITEA_CLIENT_SECRET
  Refresh Interval:         1h
  Secret Store Ref:
    Kind:  ClusterSecretStore
    Name:  global-secrets
  Target:
    Creation Policy:  Owner
    Deletion Policy:  Retain
    Name:             dex-secrets
Status:
  Conditions:
    Last Transition Time:  2024-04-18T09:48:36Z
    Message:               could not get secret data from provider
    Reason:                SecretSyncedError
    Status:                False
    Type:                  Ready
Events:
  Type     Reason        Age                    From              Message
  ----     ------        ----                   ----              -------
  Warning  UpdateFailed  4m40s (x24 over 109m)  external-secrets  error retrieving secret at .data[0], key: kanidm.dex, err: secrets "kanidm.dex" not found 

this is also affecting other deployment as well and for some reasons none of the secrets generated works when trying to login to the deployments

woodpecker          pre-install-agent-secret-check-jsqrs                     0/1     Completed                    0                75m
woodpecker          woodpecker-agent-5b6945cc7b-8c49l                        0/1     CrashLoopBackOff             19 (2m41s ago)   75m
woodpecker          woodpecker-agent-5b6945cc7b-nrmmf                        0/1     CrashLoopBackOff             19 (2m52s ago)   75m

[kikokikok]

Same problem for me, I think the
kanidm.dex key is never creqted in the global-secrets ClusterSecretStore

[khuedoan]

kandim.dex should be created by default in the post install script, could you try running make post-install manually?

[kikokikok]

Well the postscript fails when calling the reset of users with the python k8s client. It doesn't return the expected json payload on the stdout as expected which causes an error on json deserialization.
When executing with a remote ssh into the container, I see the json paylod

bash-5.2# make postinstall
make: *** No rule to make target 'postinstall'.  Stop.
bash-5.2# make post-install
Traceback (most recent call last):
  File "/home/cklat/homelab/./scripts/hacks", line 256, in <module>
    main()
  File "/home/cklat/homelab/./scripts/hacks", line 247, in main
    kanidm_login(["admin", "idm_admin"])
  File "/home/cklat/homelab/./scripts/hacks", line 158, in kanidm_login
    password = reset_kanidm_account_password(account)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cklat/homelab/./scripts/hacks", line 152, in reset_kanidm_account_password
    return json.loads(resp)['password']
           ^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 2 (char 1)

Manual bash inside the container:

kanidmd recover-account --output json admin
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: This is running as uid == 0 (root) which may be a security risk.
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: permissions on /data/server.toml may not be secure. Should be readonly to running uid. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml owned by the current uid, which may allow file permission changes. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: DB folder /data has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 INFO     ｉ [info]: Running account recovery ...
{"password":"VU29tSLcAqjccXWez12dQKhKNuPNWcJDcQ34NXK1gGGFSGwN"}