The RDMA connection count exceeds the connMaxInternodeNum parameter configured

[fan-spec]

Occasionally, when monitoring RDMA connections on a BeeGFS client using beegfs-net, the connection count surpasses the maximum allowed connections (connMaxInternodeNum) as defined in the beegfs-client.conf configuration file.

[iamjoemccormick]

Any chance you can share the following outputs while the issue is happening (with IPs and anything else potentially sensitive redacted/obfuscated)?

cat /proc/fs/beegfs/*/config
beegfs-net

Worth noting connMaxInternodeNum is per node, so it is normal to see in total the number of connections across all nodes exceeding this number.

[fan-spec]

@iamjoemccormick Thank you for you response.
Let me share more details for my case first.
The command beegfs-net show like this, when the connMaxInternodeNum = 128 set in /etc/beegfs/beegfs-client.conf and /proc/fs/beegfs/*/config
Server01 [ID: 1]
ConnectionsTCP: 107 (xx.xx.xx.xx:8003 [fallback route]); RDMA: 633 (xx.xx.xx.xx:8003) ;
Server02 [ID: 2]
....

For further detail，the client has four IPs which are not in the same subnet and we use Client Multi-Rail, the four IPs are like this
ib0 50.50.50.129
ib0 60.60.60.129
ib0 70.70.70.129
ib0 80.80.80.129

Worth noting connMaxInternodeNum is per node, so it is normal to see in total the number of connections across all nodes exceeding this number.
I think the line "ConnectionsTCP: 107 (xx.xx.xx.xx:8003 [fallback route]); RDMA: 633 (xx.xx.xx.xx:8003) ;" in beegfs-net result means one node connection, right?

[iamjoemccormick]

The line: ConnectionsTCP: 107 (xx.xx.xx.xx:8003 [fallback route]); RDMA: 633 (xx.xx.xx.xx:8003); means this client has 107 TCP connections and 633 RDMA connections to Server01. This is odd if the client has connMaxInternodeNum=128 because it shouldn't try to establish more than 128 connections to any one metadata or storage service. Connections are established once then placed in a reusable pool, and once the connMaxInternodeNum limit is reached, the client will wait for a connection to become available. Idle connections are dropped from the pool after some time and reestablished when needed.

While I haven't seen this behavior before, I had a quick look through our issue tracker and only found one similar report way back with BeeGFS 6 and OmniPath. However that turned out not to be reproducible, and the conclusion was there was likely a mismatch between the original runtime (procfs) and on-disk (/etc/beegfs) configuration that only made it seem like the limit was exceeded.

Just to sanity check, cat /proc/fs/beegfs/<CLIENT-ID>/config | grep -i connMaxInternodeNum currently shows 128 for the same <CLIENT-ID> where also cat /proc/fs/beegfs/<CLIENT-ID>/meta_nodes and/or cat /proc/fs/beegfs/<CLIENT-ID>/storage_nodes show more than 128 connections to some node(s)? Those meta_nodes and storage_nodes procfs entries are where beegfs-net gets its data from, but in case you have multiple BeeGFS mount points its good to make sure we're correlating the same runtime configuration and connection counts.

I did have a look at the latest client code that handles establishing/reusing connections and populating the statistics displayed in beegfs-net and the logic appears sound. What version of BeeGFS are you running?

[fan-spec]

The version is 7.4.4.
I will check the 'cat /proc/fs/beegfs//meta_nodes and/or cat /proc/fs/beegfs//storage_nodes' you mentioned next time I come across this issue.