Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Usage Spike when checking if refreshtokens.dex.coreos.com has been created #4019

Open
3 tasks done
TheNumberOne opened this issue Mar 3, 2025 · 1 comment · May be fixed by #4027
Open
3 tasks done

Memory Usage Spike when checking if refreshtokens.dex.coreos.com has been created #4019

TheNumberOne opened this issue Mar 3, 2025 · 1 comment · May be fixed by #4027

Comments

@TheNumberOne
Copy link

TheNumberOne commented Mar 3, 2025 •
edited
Loading

Preflight Checklist

  • I agree to follow the Code of Conduct that this project adheres to.
  • I have searched the issue tracker for an issue that matches the one I want to file, without success.
  • I am not looking for support or already pursued the available support channels without success.

Version

2.39.1

Storage Type

etcd

Installation Type

Custom Helm chart, Official container image

Expected Behavior

The RAM usage of dex shouldn't increase very much during CRD installation and should preferably stay below 64 MB.

Actual Behavior

The RAM usage of dex increases above 64MB during CRD installation, which causes it to be OOM killed in our current configuration.

Steps To Reproduce

  1. Set a memory limit of 64MB on the dex pod
  2. Cause dex to create 13531 refreshtokens
  3. Restart dex

Additional Information

I'm pretty sure this is caused by dex attempting to list all of the resources when checking if CRDs exist.

Audit logs showed no attempts to create/update the CRD before Dex crashes.

I am using a kustomized version of kubeflow's vendored dex.

A workaround is to increase/remove the memory limit.

Configuration

issuer: ...
storage:
  type: kubernetes
  config:
    inCluster: true
web:
  http: 0.0.0.0:5556
logger:
  level: "debug"
  format: text
oauth2:
  skipApprovalScreen: true
  passwordConnector: local # for deploying from mlflow
enablePasswordDB: true
staticPasswords:
- email: ...
  hash: ...
  username: ...
- email: ...
  hash: ...
  username: ...
- email: ...
  hash: ...
  username: ...
- email: ...
  hash: ...
  username: ...
- email: ...
  hash: ...
  username: ...
staticClients:
# https://github.com/dexidp/dex/pull/1664
- idEnv: OIDC_CLIENT_ID
  redirectURIs:
  - "/oauth2/callback"
  name: "Dex Login Application"
  secretEnv: OIDC_CLIENT_SECRET
- idEnv: OAUTH2_PROXY_CLIENT_ID
  redirectURIs:
  - /oauth2/callback
  name: oauth2-proxy
  secretEnv: OAUTH2_PROXY_CLIENT_SECRET
- id: kserve-client
  name: kserve-client
  secret: ...
- idEnv: INFERENCE_CLIENT_ID
  name: inference-client
  secretEnv: INFERENCE_CLIENT_SECRET
  # This is a client that can be used to programmatically authenticate to Dex with a non-private secret
- id: ...
  secretEnv: ...
  name: "..."
  public: true
connectors:
- type: google
  id: google
  name: Google
  config:
    # Connector config values starting with a "$" will read from the environment.
    clientID: $GOOGLE_CLIENT_ID
    clientSecret: $GOOGLE_CLIENT_SECRET

    # Dex's issuer URL + "/callback"
    redirectURI: ...

    # Users must have an email with the following domains
    hostedDomains:
    - ...
    - ...
    - ...

Logs

time="2025-03-03T17:19:06Z" level=info msg="Dex Version: v2.39.1, Go Version: go1.22.2, Go OS/ARCH: linux amd64"
time="2025-03-03T17:19:06Z" level=info msg="config using log level: debug"
time="2025-03-03T17:19:06Z" level=info msg="config issuer: [...]"
time="2025-03-03T17:19:06Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2025-03-03T17:19:06Z" level=info msg="creating custom Kubernetes resources"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has already been created..."
time="2025-03-03T17:19:06Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2025-03-03T17:19:06Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has already been created..."
stream closed EOF for auth/dex-6f757b7ffb-xc6zw (dex)
@nabokihms nabokihms linked a pull request Mar 5, 2025 that will close this issue
Open
@nabokihms
Copy link
Member

Opened a PR #4027
Should fix the memory issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

k8s storage: Request only one object to check if API exists deckhouse/3p-dex
2 participants
@TheNumberOne @nabokihms