module envs are missing when using SlurmSpawner

[eddy16112]

Bug description

My sbatch script is:

[I 2022-06-02 22:44:23.718 JupyterHub batchspawner:292] Spawner submitted script:
    #!/bin/bash
    #SBATCH -t 00:30:00
    #SBATCH -N 1
    env
    module load PrgEnv/GCC+OpenMPI/2021-05-27 cuda gcc openmpi
    which mpirun
    batchspawner-singleuser jupyterhub-singleuser

I am getting an error in the slurm-xxxxx.out
/var/spool/slurm/d/job242029/slurm_script: line 5: module: command not found
Looks like it can not find the module.

However, if I manually submit the script using sbatch mytest.sub, and I do not see the error.

More details

I compared the env output from the slurm.out generated by SlurmSpawner and the one generated by manual submission, and I found out the following module related envs are missing from the slurm.out generated by the spawner.

MODULEPATH=/gpfs/fs1/SHARE/Modules/Latest:/gpfs/sw/modules/all:/gpfs/fs1/SHARE/Modules/Bundles:/gpfs/fs1/SHARE/Modules/PrgEnv/GCC+OpenMPI/2021-05-27
MODULEPATH_ROOT=/gpfs/fs1/SHARE/Utils/Lmod/8.1/GCC-BASE-7.4.0/modulefiles
MODULESHOME=/gpfs/fs1/SHARE/Utils/Lmod/8.1/GCC-BASE-7.4.0
BASH_FUNC_module%%=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
BASH_FUNC_ml%%=() {  eval $($LMOD_DIR/ml_cmd "$@")
}

Is it a bug or did I miss anything when using the SlurmSpawner?

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[mahendrapaipuri]

Hey, As you rightly figured out, this is problem of the environment not being complete when batchspawner submitting the job. Under the hood, batchspawner submits the job using command sudo -E -u <target_user> sbatch --parsable <script>. You can run test this command as same user as the JupyterHub running under to see why complete environment is not being sourced. But I do not think this has to do anything with batchspawner and more with your platform configuration.