Fixed HPA rule to use more correct condition

[dharapvj]

Correct rule expression evaluation...

Current rule expression evaluation...

[samber]

LGTM.

I wonder if we need to split it into 2 metrics:

• ScalingLimited==false -> alert
• ScalingLimited==true && AbleToScale==false -> alert

WDYT ?

[dharapvj]

Sorry I did not get back to you quickly.

ScalingLimited==false -> alert

I don't think if ScalingLimited==false is a cause of concern. that just means, there is no limit on scaling right now.

Having said that - I observed, that if, we only check for ScalingLimited==true - it alerts on both side.. if it cannot scale beyond max replicas as well as when hpa cannot scale down below min-replica e.g. minReplica is 3 and hpa thinks that we only need 1 or 2 (basically - over-allocation case).

We did not want to get alerts for this over-allocation considering criticality of the system. so I combined couple of metrics to get alerted only when HPA wants to create more pods but is restricted from doing so..

My new expression looks like this:

(
  kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"YOUR_NS"} - 
  kube_horizontalpodautoscaler_status_desired_replicas{namespace=~"YOUR_NS"}
) * on (horizontalpodautoscaler,namespace)
(
  kube_horizontalpodautoscaler_status_condition{condition="ScalingLimited",namespace=~"YOUR_NS",status="true"} == 1
) == 0

Derivation of this alert expression:

a. ScalingLimited condition returns true for both when HPA hits max limit as well as min limit. we want to focus only on maxed out HPAs.
b. Difference between max and desired replicas only tells us that currently we have exhausted all possible replicas but does not indicate pressure for another replica, which is not getting fulfilled as we have reached max.

So, we take cross product of HPA scalinglimited==true and HPA where max and desired replica diff is zero. Whenever these two expressions match, their multiplication will come as zero and that's our actual alerting HPA we want to look at!

Hope this helps..

[samber]

Ok. I'm going to do the update and merge ✌️