github actions fail using authenticated github token.

[yaccad]

Hello, I have a teleport setup using helm chart for version 13.0.3. which allows access to a k8s cluster. Everything is working when I use tsh login with password authenticated users, however when using github actions to run the same helm commands that I can run locally when authenticated I'm getting the below error:

couldn't get current server API group list: Get "[https://teleport.domain.com:443/api?timeout=32s](https://teleport.domain.com/api?timeout=32s)": tls: failed to verify certificate: x509: certificate signed by unknown authority

Looking at the autdit logs as well as enabling debugging I see that the github-token user is succesfully getting authenticated.

Here is the github action debug log showing that the github-token teleport user is able to login however in the next action when trying to access the teleport cluster to run a helm upgrade it receives the error described above. The web portal is accessible on https with a valid certificate from ACME. I wonder if the api service is the issue here? Help from the community is appreciated:

INFO [TBOT]      Anonymous telemetry is not enabled. Find out more about Machine ID's anonymous telemetry at https://goteleport.com/docs/machine-id/reference/telemetry/ tbot/anonymous_telemetry.go:83
INFO [TBOT]      Attempting to generate new identity from token tbot/renew.go:479
DEBU [AUTH]      Registering node to the cluster. auth-servers:[{teleport.domain.com:443 tcp }] auth/register.go:264
DEBU [AUTH]      The first specified auth server appears to be a proxy. auth/register.go:270
INFO [AUTH]      Attempting registration via proxy server. auth/register.go:277
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.domain.com:443, insecure=false, extraHeaders=map[]) client/weblogin.go:308
DEBU [CLIENT]    Attempting https://teleport.domain.com:443/v1/webapi/host/credentials client/https_client.go:86
INFO [AUTH]      Successfully registered via proxy server. auth/register.go:284
DEBU [TBOT]      Attempting first connection using initial auth client tbot/tbot.go:407
DEBU [TBOT]      Connecting to: [{teleport.domain.com:443 tcp }]. authclient/authclient.go:57
DEBU             Attempting GET teleport.domain.com:443/webapi/find webclient/webclient.go:128
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
INFO [TBOT]      Successfully generated new bot identity, valid: after=2024-05-18T19:16:45Z, before=2024-05-18T20:17:44Z, duration=1h0m59s | kind=tls, renewable=false, disallow-reissue=false, roles=[bot-github-access], principals=[-teleport-internal-join], generation=0 tbot/tbot.go:422
DEBU [TBOT]      Storing new bot identity to [memory] tbot/tbot.go:424
DEBU [TBOT]      Writing key-cert.pub identity/identity.go:426
DEBU [TBOT]      Writing sshcacerts identity/identity.go:426
DEBU [TBOT]      Writing tlscert identity/identity.go:426
DEBU [TBOT]      Writing tlscacerts identity/identity.go:426
DEBU [TBOT]      Writing key identity/identity.go:426
DEBU [TBOT]      Writing key.pub identity/identity.go:426
DEBU [TBOT]      Writing tokenhash identity/identity.go:426
DEBU [TBOT]      Writing .write-test identity/identity.go:426
INFO [TBOT]      Beginning renewal loop: ttl=1h0m0s interval=20m0s tbot/renew.go:693
DEBU [TBOT]      Attempting to renew bot certificates... tbot/renew.go:590
DEBU [TBOT]      Attempting to establish watch for CA events tbot/ca_rotation.go:151
INFO [TBOT]      Started watching for CA rotations tbot/ca_rotation.go:173
INFO [TBOT]      Attempting to generate new identity from token tbot/renew.go:479
DEBU [AUTH]      Registering node to the cluster. auth-servers:[{teleport.domain.com:443 tcp }] auth/register.go:264
DEBU [AUTH]      The first specified auth server appears to be a proxy. auth/register.go:270
INFO [AUTH]      Attempting registration via proxy server. auth/register.go:277
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.domain.com:443, insecure=false, extraHeaders=map[]) client/weblogin.go:308
DEBU [CLIENT]    Attempting https://teleport.domain.com:443/v1/webapi/host/credentials client/https_client.go:86
INFO [AUTH]      Successfully registered via proxy server. auth/register.go:284
INFO [TBOT]      Successfully renewed bot certificates, valid: after=2024-05-18T19:16:51Z, before=2024-05-18T20:17:50Z, duration=1h0m59s | kind=tls, renewable=false, disallow-reissue=false, roles=[bot-github-access], principals=[-teleport-internal-join], generation=0 tbot/renew.go:601
DEBU [TBOT]      Connecting to: [{teleport.domain.com:443 tcp }]. authclient/authclient.go:57
DEBU             Attempting GET teleport.domain.com:443/webapi/find webclient/webclient.go:128
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [TBOT]      Auth client now using renewed credentials. tbot/renew.go:615
DEBU [TBOT]      Writing key-cert.pub identity/identity.go:426
DEBU [TBOT]      Writing sshcacerts identity/identity.go:426
DEBU [TBOT]      Writing tlscert identity/identity.go:426
DEBU [TBOT]      Writing tlscacerts identity/identity.go:426
DEBU [TBOT]      Writing key identity/identity.go:426
DEBU [TBOT]      CA watcher detected client closing. Re-watching in 1.529295465s. tbot/ca_rotation.go:132
DEBU [TBOT]      Writing key.pub identity/identity.go:426
DEBU [TBOT]      Writing tokenhash identity/identity.go:426
DEBU [TBOT]      Writing .write-test identity/identity.go:426
DEBU [TBOT]      Destination specified no roles, defaults will be requested: [github-kube-access] tbot/renew.go:171
DEBU [TBOT]      Destination specified no roles, defaults will be requested: [github-kube-access] tbot/renew.go:171
DEBU [TBOT]      Attempting to establish watch for CA events tbot/ca_rotation.go:151
INFO [TBOT]      Generated identity for Kubernetes cluster {"k8s-staging"} tbot/renew.go:422
INFO [TBOT]      Successfully renewed impersonated certificates for directory /home/runner/work/_temp/09b44199-d7[37](https://github.com/domain/service/actions/runs/9132922133/job/28923437289#step:3:38)-4182-91d3-5f4fa10e7b0a, valid: after=2024-05-18T19:16:57Z, before=2024-05-18T20:17:57Z, duration=1h1m0s | kind=tls, renewable=false, disallow-reissue=true, roles=[github-kube-access], principals=[-teleport-internal-join], generation=0 tbot/renew.go:664
DEBU [TBOT]      Writing key-cert.pub identity/identity.go:426
DEBU [TBOT]      Writing tlscert identity/identity.go:426
DEBU [TBOT]      Writing key identity/identity.go:426
DEBU [TBOT]      Writing key.pub identity/identity.go:426
DEBU [TBOT]      Writing .write-test identity/identity.go:426
DEBU [TBOT]      Connecting to: [{teleport.domain.com:443 tcp }]. authclient/authclient.go:57
INFO [TBOT]      Started watching for CA rotations tbot/ca_rotation.go:173
DEBU             Attempting GET teleport.domain.com:443/webapi/find webclient/webclient.go:128
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [TBOT]      Found OpenSSH version 8.9.1 openssh/openssh.go:205
DEBU [TBOT]      Using SSH options: sshConfigOptions: HostKeyAlgorithms will include SHA-256, SHA-512 and SHA-1 openssh/openssh.go:209
DEBU [TBOT]      Wrote identity file: [identity] config/configtemplate_identity.go:96
DEBU             Attempting GET teleport.domain.com:443/webapi/ping webclient/webclient.go:128
DEBU [TBOT]      Connecting to: [{teleport.domain.com:443 tcp }]. authclient/authclient.go:57
DEBU             Attempting GET teleport.domain.com:443/webapi/find webclient/webclient.go:128
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:195
INFO [TBOT]      Persisted certificates successfully. One-shot mode enabled so exiting. tbot/renew.go:747
##[debug]Node Action run completed with exit code 0
##[debug]KUBECONFIG='/home/runner/work/_temp/09b4[41](https://github.com/domain/service/actions/runs/5689923876/job/25133679269#step:3:42)99-d737-4182-91d3-5f4fa10e7b0a/kubeconfig.yaml'
##[debug]Finishing: Fetch credentials using Machine ID

Some additional details. This is what the helm values that deployed this teleport cluster look like:

teleport-cluster:
  enabled: true
  clusterName: "teleport.domain.com"
  kubeClusterName: "k8s-cluster"
  chartMode: "aws"
  teleportVersionOverride: "13.0.3"

  authentication:
    secondFactor: "otp"

  aws:
    region: "us-east-1"
    backendTable: "teleport.state"
    auditLogTable: "teleport.events"
    sessionRecordingBucket: "teleport-sessions"
    auditLogMirrorOnStdout: false
    backups: true
    dynamoAutoScaling: false

  # this needs to be disabled for  kubernetes >=1.23
  podSecurityPolicy:
    enabled: false

  acme: true
  acmeEmail: "<email address>"

[webvictim]

What versions of kubectl and helm are you using?

How are you running the command inside Github actions? Can you share your YAML file for the action?

It looks like tbot is running correctly and getting the identity (which is in the written KUBECONFIG path at the end)

[yaccad]

Hi @webvictim, thanks for responding I appreciate it. I ended up outputting the KUBECONFIG in the action and noticed a discrepancy on the port used to access the proxy. After changing the port that's now working. Thanks again.

[webvictim]

Glad to hear! I think we did have a bug with this in older versions of Teleport which was fixed - you might like to try upgrading to the newest version of v13 (13.4.24)

[yaccad]

Issue resolved.