Etcd nodes IO rate got higher and snapshot size got bigger #5680
-
|
I just noticed that IO is significantly increased on my RKE2 cluster with 3 servers and 3 agents. I checked I did not do any new significant workload deployments this week except upgrading Rancher from 2.8.2 to 2.8.3 and then upgrading this cluster from 1.27.12 version to v1.28.8 +rke2r1. As I remember I was doing it about 1 week ago. Maybe somebody knows what exactly reason can be which will be very helpful |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 15 replies
-
|
IO rate corresponds to create/modify/delete load from the apiserver. Snapshot size corresponds to objects in the datastore. It sounds like something caused an increase in both traffic and the number of resources. The apiserver exposes request rate and resource count metrics, I would recommend looking at these to try to determine what's changed. |
Beta Was this translation helpful? Give feedback.
-
I see some of services are using cpu constantly |
Beta Was this translation helpful? Give feedback.
-
And I see apiserver node always under the load |
Beta Was this translation helpful? Give feedback.
-
|
I don't see any info on what the events are from. Is it LH? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. A lot of put and get requests. Looks like it's related to this issue |
Beta Was this translation helpful? Give feedback.
Yes. A lot of put and get requests. Looks like it's related to this issue
longhorn/longhorn#8114