Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let JupyterHub Dashboard's "Users per node" report empty nodes as well #77

Open
consideRatio opened this issue Apr 25, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@consideRatio
Copy link
Member

The JupyterHub Dashboard's "Users per node" panel should also report nodes with 0 users on them I think, otherwise one may think the k8s cluster doesn't have idle nodes when it does.

Example situation

The dashboard panel reports very few nodes.

image

But there are plenty of nodes (nodes with labels associated with being a user node etc), just that they don't have users on them.

image

Related

@consideRatio
Copy link
Member Author

consideRatio commented Oct 24, 2023

I think this is also a bug that relates to a bug of reporting 1 user when its in practice 0, just because there is one entry at one point in time reporting 1 followed by "no value".

Here is what I observed - note how the lines change but nothing ever goes below 1

staging-not-zero

EDIT: I think this is fixed by #90 as well.

@consideRatio consideRatio added the bug Something isn't working label Oct 24, 2023
@consideRatio
Copy link
Member Author

consideRatio commented Apr 10, 2024

I think the issue is with the query, which use pod info to acquire node info, instead of using node info to then count pods. Without a user pod scheduled on a node, it isn't recognized.

sum(
    # kube_pod_info.node identifies the pod node,
    # while kube_pod_labels.node is the metrics exporter's node
    kube_pod_info{node!=""}
    * on (namespace, pod) group_left() 
  group(
    kube_pod_labels{label_app="jupyterhub", label_component="singleuser-server", namespace=~"$hub"}
  ) by (label_component, pod, namespace)
) by (node)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant