MPI Configuration

This page is for listing configuration/version information for different MPI implementations on target platforms.

Mutrino with Cray MPI (MPICH derivative)

• The man page for MPI configuration can be found by running: man intro_mpi

• For getting an informational print about MPI at startup: export MPICH_ENV_DISPLAY=1

Vortex with IBM Spectrum MPI (OpenMPI derivative)

• The man page for MPI configuration can be found by running man ompi_info and ompi_info.

• Here is a reference to the IBM Spectrum MPI manual that discusses how to tune the implementation: IBM Spectrum MPI Guide.

• You can use ompi_info --param all all to see the options. Better yet, try ompi_info --param all all -l 9 --internal.

• If you need to pass mpirun options to jsrun, use jsrun --smpiargs "<mpiarg1> <mpiarg2> ...".

• Some MCA environment variables that may be useful are:

  • export OMPI_MCA_orte_debug=true
  • export OMPI_MCA_orte_debug_verbose=9
  • export OMPI_MCA_osc_base_verbose=100

• Here is a link that discusses MCA tunnel atomics for RDMA: tunnel atomics. Enabling tunnel atomics doesn't seem to help performance of MPI_Fetch_and_op on Vortex.

• Here is a link to the administering Spectrum MPI. Has some info on how to set runtime parameters (like MCA). IBM Spectrum MPI Administration Guide

• To collect data using nsys, use jsrun to call nsys. If you only want to profile the part of execution marked by cudaProfilerStart/cudaProfilerStop, the commands are:

module load nsight-systems/2020.1.1
export KOKKOS_PROFILE_LIBRARY=/ascldap/users/mbetten/vortex/build/gcc-cuda/kokkos-tools/profiling/nvprof-connector/kp_nvprof_connector.so
jsrun <jsrun_args...> nsys profile -f true --capture-range=cudaProfilerApi --trace=cuda,nvtx,mpi --mpi-impl=openmpi -o %q{OMPI_COMM_WORLD_RANK} --kill none <app> <app_args...>

MPE Performance Analysis Tool

• See the manual here: MPE manual
• Download the frontend jumpstart from here