X509 certificates with SSO issuer

[LeJav]

Hello,

I am using argo workflow with SSO and OpenId Connect (implemented with keycloak).
It is working fine when everything is http.
But when going with https I have a problem with issuer.
server crashes because keycloak certificate authority is not known:

 msg="Get https://xxxxxxxxx/auth/realms/master/.well-known/openid-configuration: x509: certificate signed by unknown authority"

How to solve it?
I have not found any option in order to ignore SSL verification.
Will it work if I set the CA cert in configmap and mount it under /etc/ssl/certs?

Thanks for your advices.

[LeJav]

For those who have the same problem, I have solved it by setting the ca cert in a configmap and mounting it in a file under /etc/ssl/certs

[yogendramummaneni]

I am also seeing the same issue even after following the below steps:

1. Added a configmap
2. Updated the values.yaml with below and tried to install the helm chart and pod is going into CrashLoopBackOff state, logs are below:

  volumeMounts:
     - name: certificate
       mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
       #subPath: ca.crt
  volumes:
     - name: certificate
       configMap:
          name: configcert

kubectl logs -f argo-keyclock-argo-workflows-server-66d6b99ffc-n7rn2 -n argo
time="2022-07-13T15:34:32.128Z" level=info msg="config map" name=argo-keyclock-argo-workflows-workflow-controller-configmap
Error: Get "https://keycloak.xxx-xxxx.com/auth/realms/argocd/.well-known/openid-configuration": x509: certificate signed by unknown authority
Usage:
  argo server [flags]

Examples:
See https://argoproj.github.io/argo-workflows/argo-server/

Any idea what mistake I am doing here?

Thanks inadvance.

[LeJav]

How is your configmap?
On my side I have a key for the crt and I have set subPath.

[yogendramummaneni]

@LeJav Thanks for the reply!

Now I created a key using this command:

kubectl -n argo create configmap custom-root-cert --from-file=ca-cert=/home/dell/dev-vm/certs/rootCA.crt

and mounting like this inside values.yaml

  volumeMounts:
    - name: certificate
      mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
      subPath: ca-cert

  volumes:
    - name: certificate
      configMap:
        name: custom-root-cert

Still unable to mount the certificate under /etc/ssl/certs/rootCA.crt. The file rootCA.crt is being created as empty. I have checked on the overlay file system.

root@kube-cp01:~# cd /var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged# ls
bin  dev  etc  home  proc  sys  tmp  var
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged# cd etc/ssl/certs/
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs# ls
ca-certificates.crt  rootCA.crt
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs# more rootCA.crt
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs#

Any ideas what could be the wrong here?

[LeJav]

Hello,
Your configuration seems correct.
It is normal that you do not see it when looking into /var/lib/docker/overlay2/... as it is mounted.
Can you check inside the pod? kubectl exec -it .... -- sh and check if you see your certificate under /etc/ssl/certs

[sarabala1979]

Fixed in ae7eeeb I am releasing 3.1.10 today

…

Sent from my iPhone

On Sep 9, 2021, at 11:13 PM, Peter O. ***@***.***> wrote:  @LeJav did you do anything else beside mounting your CA pem file under /etc/ssl/certs? That's what I did and I am still getting this error. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[peteroruba]

@sarabala1979 thanks for your response, however, I deleted my question, because it turned out I used the wrong certificate. With the correct one the procedure as explained by @LeJav worked for me as well.

[chenbr]

did you change the deployment to use the configmap? do you have an example?

[LeJav]

I have currently no access to my configuration. I will describe next week the modifications I have done to the Helm chart in order to mount the cert configmap in the pod.

[sharadhirao]

Hi @LeJav
I have created config map as:
kubectl -n argo create configmap custom-root-cert --from-file=ca-cert=rootCA.crt

and my values.yal for volume looks like this:

volumeMounts:
- name: certificate
  mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
  subPath: ca-cert

volumes:
- name: certificate
  configMap:
    name: custom-root-cert

still I am facing the issue of 509 error. Can you help where I am going wrong?

[xmsanchez]

I got the same issue. I got this working in another environment and mounting the certs in /etc/ssl/certs is working there. I don't undertstand why it doesn't work in the new environment.

Error: Get "https://dex.pre.example.com/.well-known/openid-configuration": x509: certificate signed by unknown authority

I double checked the CA certificates, in fact I can spin up a curl container and it works perfectly with the same CAs when querying the SSO endpoint.

Any idea on how to debug this further? Log level in debug mode and enabling --verbose does not help at all.

[xmsanchez]

I fixed it. In the end it was an issue with my CA bundle. I updated it properly and it works.

As suggested in the accepted answer you must mount it at /etc/ssl/certs

[Jaydee94]

Hey @xmsanchez,

I have the same issue with my Argo workflows setup. How did you fix your ca bundle?

I've created a configmap with my custom ca in pem format and mounted it into /etc/ssl/certs/ca.crt

After a pod restart I still get the x509 certificate is signed by a unknown authority error.

Any ideas?

[Jaydee94]

Never mind. My ca bundle was also broken. The pem format was incorrect.
Works fine now.

[anushkafer]

ed a configmap with my custom ca in pem format and mounted it into /etc/ssl/certs/

@Jaydee94 Did you got fix this out?

[LeJav]

I am sorry but I am not working any more with the configuration with the certificates mounted in the pod.
I am really sorry but I cannot test and help...

[aydinseven7]

Another reminder since I had to debug a bit to get it to work:

• add the full chain
• mount it to /etc/ssl/certs/, don't add specific cert name

inside values.yaml:

server:
  volumeMounts:
  - name: intermediate-ca
    mountPath: /etc/ssl/certs/
  volumes:
  - name: intermediate-ca
    secret:
      secretName: intermediate-ca

ca-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: intermediate-ca
  namespace: argo
stringData:
  root-ca.crt: |
    CERT1
  intermediate-ca.crt: |
    CERT2
type: Opaque

[anushkafer]

Thanks a lot, but still getting same error x509: certificate signed by unknown authority. Not sure what's wrong with that.

[aydinseven7]

Just to be sure, have you added the secret with the certs into the correct namespace, then added the volume and volumemount for the secret into the values.yaml and then restarted the pod?

I specifically have not added the final cert of Keycloak into the secret, just the root and intermediate certs and it works...

[anushkafer]

Yes, created secret intermediate-ca and applied helm values file as below

argo-workflows:
  sso:
   enabled: true
.
.
.
.
.
    volumeMounts:
    - name: intermediate-ca
      mountPath: /etc/ssl/certs/
    volumes:
    - name: intermediate-ca
      secret:
        secretName: intermediate-ca

[aydinseven7]

Just to check...is the indentation in the values.yaml correct? Because in your reply it looks wrong, maybe check with a describe on the workflows server pod whether the secret actually is mounted

[anushkafer]

@aydinseven7 This is current override values.yaml and indentation seems correct. Seems Volume and Volume mount reflected correctly to workflows server pod.

argo-workflows:
  sso:
   enabled: true
  images:
    pullSecrets:
    - name: <PULL_SECRET>
  server:
    extraArgs:
      - --auth-mode=sso
    serviceAccount:
      annotations:
        workflows.argoproj.io/rbac-rule: "'<ORG>:<GROUP' in groups"
        workflows.argoproj.io/rbac-rule-precedence: "1"

    volumeMounts:
    - name: intermediate-ca
      mountPath: /etc/ssl/certs/
    volumes:
    - name: intermediate-ca
      secret:
        secretName: intermediate-ca

    ingress:
     ingressClassName: "internal"
     enabled: true
     hosts:
       - <ARGO_WORKFLOW_URL>
    sso:
      clientId:
        name: dex-static-client-credentials
        key: <CLIENT_ID>
      clientSecret:
        name: dex-static-client-credentials
        key: <CLIENT_SECRET>
      scopes:
        - groups 
      rbac:
        enabled: true
      issuer: https://<https://DEX_URL>
      redirectUrl: https://<ARGO_WORKFLOW_URL>/oauth2/callback

Secret

Pod describe

kubectl describe pod -n argo argo-workflows-server-cc45d796f-q74f4

Name:         argo-workflows-server-cc45d796f-q74f4
Namespace:    argo
Priority:     0
Node:         ip-10-132-81-40.ec2.internal/10.132.81.40
Start Time:   Tue, 25 Feb 2025 18:03:24 +0200
Labels:       app.kubernetes.io/component=server
              app.kubernetes.io/instance=argo-workflows
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=argo-workflows-server
              app.kubernetes.io/part-of=argo-workflows
              app.kubernetes.io/version=v3.2.6
              helm.sh/chart=argo-workflows-0.10.0
Annotations:  kubectl.kubernetes.io/restartedAt: 2023-07-26T11:27:43Z
Status:       Running
IP:           100.68.43.55
IPs:
  IP:           100.68.43.55
Controlled By:  ReplicaSet/argo-workflows-server-cc45d796f
Containers:
  argo-server:
    Container ID:  containerd://1fbd60e35b7f465aff133d15352d752e82fddaf4b8974ca2a0effb5c86de2f31
    Image:         quay.io/argoproj/argocli:v3.2.6
    Image ID:      quay.io/argoproj/argocli@sha256:3f7f3ad7f8e8605393e552dc1a72912b2900e02e85c1a1dc65727d104ec2d0e0
    Port:          2746/TCP
    Host Port:     0/TCP
    Args:
      server
      --configmap=argo-workflows-workflow-controller-configmap
      --auth-mode=sso
      --secure=false
    State:          Running
      Started:      Tue, 25 Feb 2025 18:04:09 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 25 Feb 2025 18:03:40 +0200
      Finished:     Tue, 25 Feb 2025 18:03:40 +0200
    Ready:          False
    Restart Count:  3
    Requests:
      cpu:      100m
      memory:   100Mi
    Readiness:  http-get http://:2746/ delay=10s timeout=1s period=20s #success=1 #failure=3
    Environment:
      IN_CLUSTER:      true
      ARGO_NAMESPACE:  argo (v1:metadata.namespace)
      BASE_HREF:       /
    Mounts:
      /etc/ssl/certs/ from intermediate-ca (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dq9zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  intermediate-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  intermediate-ca
    Optional:    false
  kube-api-access-dq9zw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true