4.x bug: `MPI_Get_accumulate` / `MPI_Rget_accumulate` with `MPI_REPLACE` hang across nodes on IB

[jeffhammond]

Open MPI: MPI_Get_accumulate / MPI_Rget_accumulate with MPI_REPLACE hang across nodes (atomic swap), while MPI_Fetch_and_op works

I am aware that RMA was rewritten in the 5.x tree, but I want to document this bug because it's rather subtle and should be available to others, even if it won't be fixed in 4.x. If it persists in 5.x, I will update the bug report.

Summary

On an InfiniBand cluster, a passive-target RMA test-and-set spinlock built from an
atomic swap hangs (livelocks) when the swap is implemented with
MPI_Get_accumulate or MPI_Rget_accumulate using MPI_REPLACE, but works
correctly when the identical swap is implemented with MPI_Fetch_and_op using
MPI_REPLACE.

The three formulations are semantically equivalent for a single-element MPI_INT
target (all are atomic read-and-replace), so all three should behave identically.
Only MPI_Fetch_and_op does. The blocking MPI_Get_accumulate case hangs as well,
so this is not specific to the request-based (R-prefixed) variant and is not
about request completion semantics.

The same reproducer passes on Intel MPI 2021.15 in all four modes, which rules out a
bug in the test program.

This was found while validating ARMCI-MPI's request-based-atomics code path
(MPI_Rget_accumulate used for ARMCI_Rmw(ARMCI_SWAP)), which deadlocks
deterministically under this Open MPI build.

Environment

Component	Version
Open MPI	4.1.9a1 (repo v4.1.5-241-g81d402c97a), from HPC-X 2.26.1
UCX	1.20.0 (multi-threaded build, ucx/mt)
Transport	HCOLL enabled; UCX over InfiniBand
OS	Rocky Linux 9.7, kernel 5.14.0-611.55.1.el9_7.x86_64
HCA	Mellanox ConnectX (mlx5), fw 16.35.4030, link_layer InfiniBand
Launch	2 nodes × 2 ranks = 4 ranks, mpirun -n 4 under Slurm

Open MPI configure line (from ompi_info):

--with-libevent=internal --enable-mpi1-compatibility --without-xpmem
--with-cuda=... --with-slurm --with-platform=contrib/platform/mellanox/optimized
--with-hcoll=... --with-ucx=... --with-ucc=...

Reproducer

A test-and-set lock: rank 0 hosts one int, initialized to 0. Each rank acquires the
lock LOOP times by atomically swapping the sentinel LOCKED (-1) into the shared
location and retrying until it reads back a non-LOCKED value; it then increments and
swaps the new value back to release. The final value must be LOOP * nprocs. The
mode argument selects the atomic primitive.

/* swaplock.c — build: mpicc -O2 -std=c99 swaplock.c -o swaplock
 * run:   mpirun -n 4 ./swaplock <mode>
 * modes: fetch_op_flush | getacc_flush | rget_flush | rget_noflush            */
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define LOCKED -1
static int LOOP = 100;
int main(int argc, char** argv){
  MPI_Init(&argc,&argv);
  int me,np; MPI_Comm_rank(MPI_COMM_WORLD,&me); MPI_Comm_size(MPI_COMM_WORLD,&np);
  const char* mode = (argc>1)?argv[1]:"rget_flush";
  int *base=NULL; MPI_Win win;
  MPI_Win_allocate((me==0)?sizeof(int):0, sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &base, &win);
  if(me==0) *base=0;
  MPI_Win_lock_all(0, win);
  MPI_Barrier(MPI_COMM_WORLD);
  double t0=MPI_Wtime();
  for(int i=0;i<LOOP;i++){
    int val=LOCKED, out;
    do{
      if(!strcmp(mode,"rget_flush")){
        MPI_Request r; MPI_Rget_accumulate(&val,1,MPI_INT,&out,1,MPI_INT,0,0,1,MPI_INT,MPI_REPLACE,win,&r);
        MPI_Wait(&r,MPI_STATUS_IGNORE); MPI_Win_flush(0,win);
      } else if(!strcmp(mode,"rget_noflush")){
        MPI_Request r; MPI_Rget_accumulate(&val,1,MPI_INT,&out,1,MPI_INT,0,0,1,MPI_INT,MPI_REPLACE,win,&r);
        MPI_Wait(&r,MPI_STATUS_IGNORE);
      } else if(!strcmp(mode,"getacc_flush")){
        MPI_Get_accumulate(&val,1,MPI_INT,&out,1,MPI_INT,0,0,1,MPI_INT,MPI_REPLACE,win); MPI_Win_flush(0,win);
      } else { /* fetch_op_flush */
        MPI_Fetch_and_op(&val,&out,MPI_INT,0,0,MPI_REPLACE,win); MPI_Win_flush(0,win);
      }
      val=out;
    }while(val==LOCKED);
    val++;
    if(!strcmp(mode,"rget_flush")||!strcmp(mode,"rget_noflush")){
      MPI_Request r; MPI_Rget_accumulate(&val,1,MPI_INT,&out,1,MPI_INT,0,0,1,MPI_INT,MPI_REPLACE,win,&r);
      MPI_Wait(&r,MPI_STATUS_IGNORE); if(strcmp(mode,"rget_noflush")) MPI_Win_flush(0,win);
    } else if(!strcmp(mode,"getacc_flush")){
      MPI_Get_accumulate(&val,1,MPI_INT,&out,1,MPI_INT,0,0,1,MPI_INT,MPI_REPLACE,win); MPI_Win_flush(0,win);
    } else {
      MPI_Fetch_and_op(&val,&out,MPI_INT,0,0,MPI_REPLACE,win); MPI_Win_flush(0,win);
    }
  }
  MPI_Barrier(MPI_COMM_WORLD);
  MPI_Win_unlock_all(win);
  if(me==0) printf("mode=%-13s final=%d expected=%d %s (%.2fs)\n", mode, *base, LOOP*np,
                   (*base==LOOP*np)?"OK":"WRONG", MPI_Wtime()-t0);
  MPI_Win_free(&win); MPI_Finalize(); return 0;
}

Observed behavior (Open MPI 4.1.9a1 / HPC-X 2.26.1, 4 ranks / 2 nodes)

[fetch_op_flush]  final=400 expected=400 OK (0.01s)
[getacc_flush]    -> HANG (killed after 45s timeout)
[rget_flush]      -> HANG (killed after 45s timeout)
[rget_noflush]    -> HANG (killed after 45s timeout)

• MPI_Fetch_and_op(MPI_REPLACE) — completes instantly, correct result.
• MPI_Get_accumulate(MPI_REPLACE) (blocking) + MPI_Win_flush — hangs.
• MPI_Rget_accumulate(MPI_REPLACE) + MPI_Wait + MPI_Win_flush — hangs.
• MPI_Rget_accumulate(MPI_REPLACE) + MPI_Wait only — hangs.

Adding an explicit MPI_Win_flush after completion does not help, so this is not
about origin-vs-target completion of the request; the atomic itself does not appear to
make forward progress / return a correct fetched value across nodes.

Expected behavior

All four modes should complete with final=400 (LOOP=100 × 4 ranks), since for a
single MPI_INT element MPI_Fetch_and_op, MPI_Get_accumulate, and
MPI_Rget_accumulate with MPI_REPLACE are equivalent atomic read-and-replace
operations.

Cross-check: Intel MPI 2021.15 (same nodes, same source)

[fetch_op_flush]  final=400 expected=400 OK (0.00s)
[getacc_flush]    final=400 expected=400 OK (0.00s)
[rget_flush]      final=400 expected=400 OK (0.00s)
[rget_noflush]    final=400 expected=400 OK (0.00s)

All modes pass, confirming the reproducer is correct MPI code and the hang is specific
to this Open MPI / UCX build.

Notes / things to try for triage

• Single-node (mpirun -n 4 on one node) vs multi-node, to isolate the UCX transport
  path (shared memory vs IB) — the failure was observed across 2 nodes.
• Whether MPI_Get_accumulate with MPI_SUM (fetch-and-add) also fails, or only
  MPI_REPLACE. (In the ARMCI test suite, fetch-and-add via Rget_accumulate
  completed and only the swap/REPLACE path deadlocked, suggesting MPI_REPLACE
  specifically.)
• UCX atomics mode: UCX_ATOMIC_MODE=guess|cpu|device — MPI_Fetch_and_op likely
  maps to a hardware atomic path while Get_accumulate(REPLACE) takes a different
  (broken) path.
• Non-mt UCX build.