BeeGFS slow when using spack on it.

[grisuthedragon]

We have the following problem in our setup. We host two beegfs storage servers with both 2x32 TB storage and 1 TB meta data storage on SSDs. Both are connected via OmniPATH/RDMA to the clients. As long as no modules (tcl or lmod) defining LD_LIBRARY_PATH are loaded everything works fine. If we load for example a gcc built with spack on the beegfs, we get

$ time ls /beegfs/home/user1/tmp 
.....

real    0m0.124s
user    0m0.000s
sys     0m0.008s

$ module load compiler/gcc/13.3.0   ( or spack load [email protected]) 
$ time ls /beegfs/home/user1/tmp 
.....
real    0m3.900s
user    0m0.000s
sys    0m0.062s

with around 30 files in the directory.

We figured out, that one difference is, that your user have set a relatively large LD_LIBRARY_PATH (around 20 entries pointing to directories in the beegfs. That means beegfs gets very slow, when using spack. But also binaries, that contain a long list of rpaths pointing to locations on the beegfs are slown down in start up. That makes using compilers, frameworks etc. installed from spack on beegfs nearly impossible. For example configuring a project with cmake on a laptop take around 60s on the beegfs it takes up to an hour.