Device to Device transfers don't work with OpenMPI + LinkX provider on AMD GPUs

[angainor]

OpenMPI 5.0.6 with shm+cxi:lnx fails to perform Device - Device transfers on LUMI system (AMD GPUs) with OSU benchmark. Host - Host transfers work as expected for intra- and inter-node transfers. For Device - Device transfers OpenMPI fails with

export FI_LNX_PROV_LINKS=shm+cxi
mpirun --mca opal_common_ofi_provider_include "shm+cxi:lnx" -np 2 -map-by numa ./osu_bibw -m 131072: D D

# OSU MPI-ROCM Bi-Directional Bandwidth Test v7.4
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
--------------------------------------------------------------------------
Open MPI failed to register your buffer.
This error is fatal, your job will abort

  Buffer Type: rocm
  Buffer Address: 0x154beaa00000
  Buffer Length: 131072
  Error: Required key not available (4294967030)
--------------------------------------------------------------------------

@hppritcha identified the problem to be related to #11076. There was a fix for this issue in #12290, but it was not merged to the 5.x branch.

[jsquyres]

AMD: Can you reply?

[edgargabriel]

I will need the help here from @hppritcha and @amirshehataornl (who developed the linkx provider in libfabric), since I am not thaat familiar with this code path. If its as simple as backport PR #12290, then it shouldn't be a challenge.

[hppritcha]

This should be assigned to @naughtont3

[jsquyres]

Thanks @edgargabriel. Might want to look into this soon, so that it can get into 5.0.7 final, if possible.

[tmh97]

I am hitting the same issue with cuda buffers when doing:

export FI_LNX_PROV_LINKS=shm+opx
mpirun --mca opal_common_ofi_provider_include "shm+opx:lnx"

Side note:
I tried the workaround that was applied to cxi here, and replaced cxi with shm on the referenced lines. I was able to run osu_bw D D intranode, but my run only got through 1 byte, then segfaulted.

[hppritcha]

Does using opx provider alone work for D to D? Why would you mix opx with shm. It already should be giving good intra node messaging performance.

[edgargabriel]

Created the backport for #12290 with #13090

[tmh97]

@hppritcha OPX provider alone works for D to D intranode, we have our own SHM fifo implementation. But, I was curious if SHM provider has better D to D intranode. It would be convenient for OPX to hook into a shared memory implementation that gives us IPC support and other nice things.