Server threads on 100% CPU

[mhardcastle]

We run a BeeGFS installation with around 6 PB storage spread over 30 servers, running over Infiniband.

We recently upgraded to 7.4.4 on Rocky Linux 9.4, kernel version 5.14.0-427.33.1.el9_4.x86_64, and after a couple of weeks of operation are seeing a few servers that are sitting at 100% CPU use on 12 of the beegfs-storage threads. iotop suggests that the processes are constantly writing to the storage disk at a low level.

I'm sure this would go away on restarting beegfs-storage, but that would affect running jobs. And it would be useful to know whether there is something we could do that would prevent it happening in future.

Any suggestions?

[iamjoemccormick]

HI @mhardcastle ,

I'm not aware of any specific changes with 7.4.4 likely to cause a low level of writes to the storage targets.

When I hear some BeeGFS threads are at 100% CPU use my first thought is that the tuneUseAggressiveStreamPoll option is enabled.

Otherwise any chance there is something actually writing to the storage servers at a low level? You might have a look at beegfs-ctl --clientstats --nodetype=meta --names and beegfs-ctl --clientstats --nodetype=storage --names and make sure there is nothing unexpected.

Probably a long shot it would be related, but do any of your clients have tuneFileCacheType=native set?