hdf5 mpi tests fail/hang

[ashterenli]

hdf5 (and netcdf) are widely used libs in production codes, e.g. in weather and climate.

Before using hdf5 libs, built with ompi, in production, I'd like to make sure the hdf5 mpi tests pass.

I guess the issue might as well be with hdf5, but since the tests pass with the intel mpi, I'll open the ticket here.

I'm seeing the same behaviour with ompi 5.0.9 and with recent master branch commits:

pmix/99-2026-01-18-1eb961d0
prrte/99-2026-01-18-97e683de
openmpi-ifort/99-2026-01-15-a5a7068b

The modules are mine. The version has the commit tag, and the date.

I'm building hdf5 1.14.6 with:

COMMON_FLAGS="-O3 -march=core-avx2"

./configure \
CC=cc \
CFLAGS="-std=c11 -Wbad-function-cast ${COMMON_FLAGS}" \
FC=ftn \
FCFLAGS="${COMMON_FLAGS}" \
--enable-parallel \
--enable-hl \
--enable-fortran \
--with-zlib=...

where ftn and cc are my wrappers for Intel oneapi C compiler and Intel ifort Fortran compiler.

The mpi tests seem to hang (or run very slow) after this:

============================
Test log for t_mpi
============================
*** Hint ***
You can use environment variable HDF5_PARAPREFIX to run parallel test files in a
different directory or to add file type prefix. e.g.,
   HDF5_PARAPREFIX=pfs:/PFS/user/me
   export HDF5_PARAPREFIX
*** End of Hint ***
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=tst5-hbv4-2
MPI-process 2. hostname=tst5-hbv4-2
MPI-process 4. hostname=tst5-hbv4-2
MPI-process 1. hostname=tst5-hbv4-2
MPI-process 3. hostname=tst5-hbv4-2
MPI-process 5. hostname=tst5-hbv4-2
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type

which starts here:
https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/testpar/t_mpi.c#L1159

and calls test_mpio_gb_file:

https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/testpar/t_mpi.c#L190

ps ax shows:

2494942 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi
2494943 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi
2494944 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi
2494951 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi
2494959 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi
2494970 ?        Rl     1:17 /dev/shm/SPE/1769088574/hdf5-1.14.6/testpar/.libs/t_mpi

Attaching gdb to a randomly chosen process of the above, I see some progress,
i.e. the bt changes with time.

So I might see

(gdb) bt
#0  uct_ib_mlx5_poll_cq (iface=0xa88b010, cq=<optimized out>, poll_flags=2, check_cqe_cb=<optimized out>) at ib_mlx5.inl:168
#1  uct_rc_mlx5_iface_poll_rx_cq (iface=0xa88b010, poll_flags=2) at rc/rc_mlx5.inl:277
#2  uct_rc_mlx5_iface_common_poll_rx (iface=0xa88b010, poll_flags=2) at rc/rc_mlx5.inl:1471
#3  uct_rc_mlx5_iface_progress (arg=0xa88b010, flags=2) at rc/rc_mlx5_iface.c:111
#4  uct_rc_mlx5_iface_progress_cyclic (arg=0xa88b010) at rc/rc_mlx5_iface.c:121
#5  0x0000068f8ba9fc66 in ucs_callbackq_dispatch (cbq=<optimized out>)
    at /dev/shm/SPE/1768926075/ucx-1.20.0/src/ucs/datastruct/callbackq.h:215
#6  uct_worker_progress (worker=<optimized out>) at /dev/shm/SPE/1768926075/ucx-1.20.0/src/uct/api/uct.h:2824
#7  ucp_worker_progress (worker=0xa6eca60) at core/ucp_worker.c:3071
#8  0x0000068f8bb93a6d in opal_progress ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libopen-pal.so.0
#9  0x0000068f8dc61365 in ompi_request_default_wait_all ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#10 0x0000068f8dd08f49 in ompi_coll_base_barrier_intra_basic_linear ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#11 0x0000068f8dce25c4 in mca_common_ompio_file_close ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#12 0x0000068f8ddd37f7 in mca_io_ompio_file_close ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#13 0x0000068f8dc5214e in file_destructor ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#14 0x0000068f8dc52e34 in ompi_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#15 0x0000068f8dc8f23f in PMPI_File_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#16 0x0000000000405275 in ?? ()
#17 0x0000068f8d827efb in __libc_start_call_main () from /usr/lib/libc.so.6
#18 0x0000068f8d827fbb in __libc_start_main () from /usr/lib/libc.so.6
#19 0x0000000000404b75 in ?? ()
(gdb)

or

(gdb) bt
#0  uct_mm_iface_poll_fifo (iface=0x1be4b3a0) at sm/mm/base/mm_iface.c:348
#1  uct_mm_iface_progress (tl_iface=0x1be4b3a0) at sm/mm/base/mm_iface.c:410
#2  0x00000a5a9a62cc66 in ucs_callbackq_dispatch (cbq=<optimized out>)
    at /dev/shm/SPE/1768926075/ucx-1.20.0/src/ucs/datastruct/callbackq.h:215
#3  uct_worker_progress (worker=<optimized out>) at /dev/shm/SPE/1768926075/ucx-1.20.0/src/uct/api/uct.h:2824
#4  ucp_worker_progress (worker=0x1bde98a0) at core/ucp_worker.c:3071
#5  0x00000a5a9a720a6d in opal_progress ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libopen-pal.so.0
#6  0x00000a5a9c9d1e65 in mca_pml_ucx_recv ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#7  0x00000a5a9c8a1fc9 in ompi_coll_base_barrier_intra_basic_linear ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#8  0x00000a5a9c9f89e4 in mca_sharedfp_sm_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#9  0x00000a5a9c87b211 in mca_common_ompio_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#10 0x00000a5a9c96c7bf in mca_io_ompio_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#11 0x00000a5a9c969ebf in mca_io_base_file_select ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#12 0x00000a5a9c7ebdf4 in ompi_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#13 0x00000a5a9c82823f in PMPI_File_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#14 0x0000000000405275 in ?? ()
#15 0x00000a5a9c427efb in __libc_start_call_main () from /usr/lib/libc.so.6
#16 0x00000a5a9c427fbb in __libc_start_main () from /usr/lib/libc.so.6
#17 0x0000000000404b75 in ?? ()

or

(gdb) bt
#0  0x00000cdfb5ac507e in uct_rc_mlx5_iface_progress_cyclic (arg=0x3bea9010) at rc/rc_mlx5_iface.c:120
#1  0x00000cdfb6644c66 in ucs_callbackq_dispatch (cbq=<optimized out>)
    at /dev/shm/SPE/1768926075/ucx-1.20.0/src/ucs/datastruct/callbackq.h:215
#2  uct_worker_progress (worker=<optimized out>) at /dev/shm/SPE/1768926075/ucx-1.20.0/src/uct/api/uct.h:2824
#3  ucp_worker_progress (worker=0x3bd0b4e0) at core/ucp_worker.c:3071
#4  0x00000cdfb89dde91 in mca_pml_ucx_recv ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#5  0x00000cdfb88adfc9 in ompi_coll_base_barrier_intra_basic_linear ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#6  0x00000cdfb88875c4 in mca_common_ompio_file_close ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#7  0x00000cdfb89787f7 in mca_io_ompio_file_close ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#8  0x00000cdfb87f714e in file_destructor ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#9  0x00000cdfb87f7e34 in ompi_file_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#10 0x00000cdfb883423f in PMPI_File_open ()
   from /data/ashterenli/spe-tst5/install/u_intel/2025.1.0.666/openmpi-ifort-99-2026-01-15-a5a7068b/lib/libmpi.so.0
#11 0x0000000000405275 in ?? ()
#12 0x00000cdfb8427efb in __libc_start_call_main () from /usr/lib/libc.so.6
#13 0x00000cdfb8427fbb in __libc_start_main () from /usr/lib/libc.so.6
#14 0x0000000000404b75 in ?? ()

but after maybe 20 min, the test exists with:

prterun noticed that process rank 2 with PID ... on node tst5-hbv4-2 exited on
signal 14 (Alarm clock).

With intelmpi (same oneapi compiler + ifort) the mpi tests pass in ~1 min, or less.

I suspect these test failures might be related to weird hangs and crashes in real applications using
the same toolchain.
So I'm keen to resolve these.

Maybe somebody can reproduce building hdf5
Thank you

[ashterenli]

Another data point -- I'm seeing the same problem with nvhpc/25.11 which has a much older openmpi:

                 Package: Open MPI qa@sky3 Distribution
                Open MPI: 4.1.9a1
  Open MPI repo revision: v4.1.5-232-gad48c462ff
   Open MPI release date: Unreleased developer copy
                Open RTE: 4.1.9a1
  Open RTE repo revision: v4.1.5-232-gad48c462ff
   Open RTE release date: Unreleased developer copy
                    OPAL: 4.1.9a1
      OPAL repo revision: v4.1.5-232-gad48c462ff
       OPAL release date: Unreleased developer copy
                 MPI API: 3.1.0
            Ident string: 4.1.9a1
                  Prefix: /data/ashterenli/spe-tst5/install/nvhpc-25.11/Linux_x86_64/25.11/comm_libs/13.0/hpcx/hpcx-2.25.1/ompi
 Configured architecture: x86_64-pc-linux-gnu
          Configure host: sky3
           Configured by: qa
           Configured on: Fri Nov 21 17:24:27 UTC 2025
          Configure host: sky3
  Configure command line: 'CC=gcc' 'CXX=g++' 'FC=nvfortran'
                          'LDFLAGS=-Wl,-rpath-link=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/ucx/lib
                          -Wl,-rpath-link=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/hcoll/lib'
                          '--with-platform=../contrib/platform/nvhpc/optimized'
                          '--enable-mpi1-compatibility'
                          '--with-libevent=internal' '--without-xpmem'
                          '--without-cray-xpmem' '--with-slurm'
                          '--with-cuda=/proj/cuda/13.0/Linux_x86_64'
                          '--with-hcoll=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/hcoll'
                          '--with-ucc=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/ucc'
                          '--with-ucx=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/ucx'
                          '--prefix=/proj/nv/libraries/Linux_x86_64/25.11/hpcx-13/289927-rel-1/comm_libs/13.0/hpcx/hpcx-2.25.1/ompi'

where the stacks might look like:

(gdb) bt
#0  0x00000782b7610f2f in ucc_tl_shm_fanout_signal (task=0x2d77da80, tree=0x2d79b380, seg=<optimized out>, team=0x2c9166f0)
    at barrier/../fanout/fanout.h:30
#1  ucc_tl_shm_barrier_progress (coll_task=<optimized out>) at barrier/barrier.c:66
#2  0x00000782d901336b in ucc_pq_st_progress (pq=0x2d054bd0) at core/ucc_progress_queue_st.c:32
#3  0x00000782d900df57 in ucc_progress_queue (pq=<optimized out>) at core/ucc_progress_queue.h:48
#4  ucc_context_progress (context=0x2ced2600) at core/ucc_context.c:1033
#5  0x00000782d9402567 in mca_coll_ucc_progress () at ../../../../../ompi/mca/coll/ucc/coll_ucc_module.c:62
#6  0x00000782fa2338fc in opal_progress () at ../../opal/runtime/opal_progress.c:231
#7  0x00000782d940500f in coll_ucc_req_wait (req=0x2d77da80) at ../../../../../ompi/mca/coll/ucc/coll_ucc_common.h:69
#8  mca_coll_ucc_barrier (comm=0x2d189290, module=0x2ca2ffd0) at ../../../../../ompi/mca/coll/ucc/coll_ucc_barrier.c:34
#9  0x000007829c607125 in mca_common_ompio_file_close (ompio_fh=ompio_fh@entry=0x2d789a00)
    at ../../../../../ompi/mca/common/ompio/common_ompio_file_open.c:262
#10 0x000007829ca04c2c in mca_io_ompio_file_close (fh=<optimized out>) at ../../../../../ompi/mca/io/ompio/io_ompio_file_open.c:97
#11 0x00000782fce497fc in file_destructor (file=0x2d197ba0) at ../../ompi/file/file.c:282
#12 0x00000782fce49d89 in opal_obj_run_destructors (object=0x2d197ba0) at ../../opal/class/opal_object.h:483
#13 ompi_file_open (comm=comm@entry=0x782fd11ae40 <ompi_mpi_comm_world>, filename=filename@entry=0x413e60 "MPItest.h5",
    amode=amode@entry=9, info=info@entry=0x782fd122d60 <ompi_mpi_info_null>, fh=fh@entry=0x7ffd2d3a6988) at ../../ompi/file/file.c:136
#14 0x00000782fce70867 in PMPI_File_open (fh=0x7ffd2d3a6988, info=0x782fd122d60 <ompi_mpi_info_null>, amode=9,
    filename=0x413e60 "MPItest.h5", comm=0x782fd11ae40 <ompi_mpi_comm_world>) at pfile_open.c:109
#15 PMPI_File_open (comm=0x782fd11ae40 <ompi_mpi_comm_world>, filename=0x413e60 "MPItest.h5", amode=9,
    info=0x782fd122d60 <ompi_mpi_info_null>, fh=0x7ffd2d3a6988) at pfile_open.c:52
#16 0x0000000000404c0b in ?? ()
#17 0x00000782faa27efb in __libc_start_call_main () from /usr/lib/libc.so.6
#18 0x00000782faa27fbb in __libc_start_main () from /usr/lib/libc.so.6
#19 0x0000000000404825 in ?? ()

or

(gdb) bt
#0  0x000010ce1de10f26 in ucc_tl_shm_fanout_signal (task=0x6a5e140, tree=0x6a7b6a0, seg=<optimized out>, team=0x5b796f0)
    at barrier/../fanout/fanout.h:31
#1  ucc_tl_shm_barrier_progress (coll_task=<optimized out>) at barrier/barrier.c:66
#2  0x000010ce4821336b in ucc_pq_st_progress (pq=0x62cb770) at core/ucc_progress_queue_st.c:32
#3  0x000010ce4820df57 in ucc_progress_queue (pq=<optimized out>) at core/ucc_progress_queue.h:48
#4  ucc_context_progress (context=0x61353c0) at core/ucc_context.c:1033
#5  0x000010ce48602567 in mca_coll_ucc_progress () at ../../../../../ompi/mca/coll/ucc/coll_ucc_module.c:62
#6  0x000010ce60c338fc in opal_progress () at ../../opal/runtime/opal_progress.c:231
#7  0x000010ce4860500f in coll_ucc_req_wait (req=0x6a5e140) at ../../../../../ompi/mca/coll/ucc/coll_ucc_common.h:69
#8  mca_coll_ucc_barrier (comm=0x63ec520, module=0x5c92fa0) at ../../../../../ompi/mca/coll/ucc/coll_ucc_barrier.c:34
#9  0x000010ce03207125 in mca_common_ompio_file_close (ompio_fh=ompio_fh@entry=0x6a6a100)
    at ../../../../../ompi/mca/common/ompio/common_ompio_file_open.c:262
#10 0x000010ce03604c2c in mca_io_ompio_file_close (fh=<optimized out>) at ../../../../../ompi/mca/io/ompio/io_ompio_file_open.c:97
#11 0x000010ce638497fc in file_destructor (file=0x63fae30) at ../../ompi/file/file.c:282
#12 0x000010ce63849d89 in opal_obj_run_destructors (object=0x63fae30) at ../../opal/class/opal_object.h:483
#13 ompi_file_open (comm=comm@entry=0x10ce63b1ae40 <ompi_mpi_comm_world>, filename=filename@entry=0x413e60 "MPItest.h5",
    amode=amode@entry=9, info=info@entry=0x10ce63b22d60 <ompi_mpi_info_null>, fh=fh@entry=0x7fffd4c85a58) at ../../ompi/file/file.c:136
#14 0x000010ce63870867 in PMPI_File_open (fh=0x7fffd4c85a58, info=0x10ce63b22d60 <ompi_mpi_info_null>, amode=9,
    filename=0x413e60 "MPItest.h5", comm=0x10ce63b1ae40 <ompi_mpi_comm_world>) at pfile_open.c:109
#15 PMPI_File_open (comm=0x10ce63b1ae40 <ompi_mpi_comm_world>, filename=0x413e60 "MPItest.h5", amode=9,
    info=0x10ce63b22d60 <ompi_mpi_info_null>, fh=0x7fffd4c85a58) at pfile_open.c:52
#16 0x0000000000404c0b in ?? ()
#17 0x000010ce61427efb in __libc_start_call_main () from /usr/lib/libc.so.6
#18 0x000010ce61427fbb in __libc_start_main () from /usr/lib/libc.so.6
#19 0x0000000000404825 in ?? ()

and again, after about 20 min, the test fails.

[edgargabriel]

@ashterenli thank you for the bug report. I assume your Open MPI 5.0.x was configured similarly to the line that you show for 4.1.9? And what is the file system that you are using to run your tests on? How many processes are you using to run t_mpi?

I run the test many times on my workstation, and it seems to pass without issues for me:

egabriel@amdhome:~/hdf5-1.14.4.3/testpar$ mpirun -np 4 ./t_mpi
*** Hint ***
You can use environment variable HDF5_PARAPREFIX to run parallel test files in a
different directory or to add file type prefix. e.g.,
   HDF5_PARAPREFIX=pfs:/PFS/user/me
   export HDF5_PARAPREFIX
*** End of Hint ***
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=amdhome
MPI-process 2. hostname=amdhome
MPI-process 1. hostname=amdhome
MPI-process 3. hostname=amdhome
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type
MPIO GB file write test MPItest.h5
MPIO GB file read test MPItest.h5
Test if MPI_File_get_size works correctly with MPItest.h5
--------------------------------
Proc 0: *** MPIO independent overlapping writes...
--------------------------------
--------------------------------
Proc 0: *** MPIO complicated derived datatype test...
--------------------------------
--------------------------------
Proc 0: *** MPIO special collective io test...
--------------------------------
===================================
MPI tests finished with no errors
===================================
egabriel@amdhome:~/hdf5-1.14.4.3/testpar$ mpirun -np 6 ./t_mpi
*** Hint ***
You can use environment variable HDF5_PARAPREFIX to run parallel test files in a
different directory or to add file type prefix. e.g.,
   HDF5_PARAPREFIX=pfs:/PFS/user/me
   export HDF5_PARAPREFIX
*** End of Hint ***
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=amdhome
MPI-process 1. hostname=amdhome
MPI-process 2. hostname=amdhome
MPI-process 5. hostname=amdhome
MPI-process 3. hostname=amdhome
MPI-process 4. hostname=amdhome
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type
MPIO GB file write test MPItest.h5
MPIO GB file read test MPItest.h5
Test if MPI_File_get_size works correctly with MPItest.h5
--------------------------------
Proc 0: *** MPIO independent overlapping writes...
--------------------------------
--------------------------------
Proc 0: *** MPIO complicated derived datatype test...
--------------------------------
--------------------------------
Proc 0: *** MPIO special collective io test...
--------------------------------
===================================
MPI tests finished with no errors
===================================
egabriel@amdhome:~/hdf5-1.14.4.3/testpar$ mpirun -np 8 ./t_mpi
*** Hint ***
You can use environment variable HDF5_PARAPREFIX to run parallel test files in a
different directory or to add file type prefix. e.g.,
   HDF5_PARAPREFIX=pfs:/PFS/user/me
   export HDF5_PARAPREFIX
*** End of Hint ***
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=amdhome
MPI-process 1. hostname=amdhome
MPI-process 2. hostname=amdhome
MPI-process 5. hostname=amdhome
MPI-process 6. hostname=amdhome
MPI-process 4. hostname=amdhome
MPI-process 7. hostname=amdhome
MPI-process 3. hostname=amdhome
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type
MPIO GB file write test MPItest.h5
MPIO GB file read test MPItest.h5
Test if MPI_File_get_size works correctly with MPItest.h5
--------------------------------
Proc 0: *** MPIO independent overlapping writes...
--------------------------------
--------------------------------
Proc 0: *** MPIO complicated derived datatype test...
--------------------------------
--------------------------------
Proc 0: *** MPIO special collective io test...
--------------------------------
===================================
MPI tests finished with no errors
===================================

So we might have to identifty what is the difference between my setup and yours. I would probably start by excluding as many optional components as possible, e.g. for not using ucc or hcoll by setting --mca coll ^ucc,hcoll

Open MPI builds and runs btw some hdf5 tests as part of every pull request in the CI, see
https://github.com/open-mpi/ompi/blob/main/.github/workflows/hdf5-tests.yaml#L46

However, the test that you mention is not among them, we only run testhpdf5, t_shapesame, and t_filters_parallel

[edgargabriel]

I also wanted to add that running t_mpi on my workstation takes only a couple of seconds

[ashterenli]

Sorry about the delay.
I tried few more thing and collected more data.

I can confirm that hdf5-1.14.6/testpar/t_mpi hangs for me
with openmpi 5.0.9, when using both intel (2025.3.1.55) and gcc (15.2.0) compilers.

Answering your questions

1. gcc compiler, ompi was configured like this:

$ ompi_info
                 Package: Open MPI anton.shterenlikht@tst5-hbv4-1
                          Distribution
                Open MPI: 5.0.9
  Open MPI repo revision: v5.0.9
   Open MPI release date: Oct 30, 2025
                 MPI API: 3.1.0
            Ident string: 5.0.9
                  Prefix: /data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/openmpi-5.0.9
 Configured architecture: x86_64-pc-linux-gnu
           Configured by: anton.shterenlikht
           Configured on: Mon Feb 16 13:21:45 UTC 2026
          Configure host: tst5-hbv4-1
  Configure command line: 'CC=gcc' 'CFLAGS=-march=znver4 -O3
                          -Wno-error=int-conversion
                          -Wno-error=incompatible-pointer-types' 'CXX=g++'
                          'CXXFLAGS=-march=znver4 -O3' 'FC=gfortran'
                          'FCFLAGS=-march=znver4 -O3'
                          '--enable-mpi-ext=affinity,pcollreq'
                          '--enable-mpi-fortran=all'
                          '--enable-mpi1-compatibility' '--enable-static'
                          '--with-cma'
                          '--with-libevent=/data/ashterenli/pe-hbv4/1.0/install/libevent-2.1.12-stable'
                          '--with-hwloc=/data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/hwloc-2.12.2'
                          '--with-ucx=/data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/ucx-1.20.0'
                          '--with-zlib=/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1'
                          '--without-cuda'
                          '--with-pmix=/data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/pmix-5.0.9'
                          '--with-prrte=/data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/prrte-3.0.12'
                          '--with-slurm' '--without-xpmem'
                          '--prefix=/data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/openmpi-5.0.9'
                Built by: anton.shterenlikht
                Built on: Mon Feb 16 13:27:03 UTC 2026
              Built host: tst5-hbv4-1
              C bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the gfortran compiler and/or Open
                          MPI, does not support the following: array
                          subsections, direct passthru (where possible) to
                          underlying Open MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: gcc
     C compiler absolute: /bin/gcc
  C compiler family name: GNU
      C compiler version: 15.2.0
            C++ compiler: g++
   C++ compiler absolute: /bin/g++
           Fort compiler: gfortran
       Fort compiler abs: /bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, Event lib: yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
          MPI extensions: affinity
 Fault Tolerance support: yes
          FT MPI support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
         MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.9)
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.9)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.9)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                 MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: uct (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.9)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.9)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.9)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v5.0.9)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.9)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                MCA smsc: xpmem (MCA v2.1.0, API v1.0.0, Component v5.0.9)
             MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.9)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                 MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
                          v5.0.9)
                MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
                          v5.0.9)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v5.0.9)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.9)
                 MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
                          v5.0.9)
                 MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.9)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.9)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v5.0.9)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
$

2. intel compiler, openmpi 5.0.9 was configured like this:

$ ompi_info
                 Package: Open MPI anton.shterenlikht@tst5-hbv4-1
                          Distribution
                Open MPI: 5.0.9
  Open MPI repo revision: v5.0.9
   Open MPI release date: Oct 30, 2025
                 MPI API: 3.1.0
            Ident string: 5.0.9
                  Prefix: /data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/openmpi-5.0.9
 Configured architecture: x86_64-pc-linux-gnu
           Configured by: anton.shterenlikht
           Configured on: Mon Feb 16 13:28:22 UTC 2026
          Configure host: tst5-hbv4-1
  Configure command line: 'CC=icx' 'CFLAGS=-march=core-avx2 -O3
                          -Wno-error=int-conversion
                          -Wno-error=incompatible-pointer-types' 'CXX=icpx'
                          'CXXFLAGS=-march=core-avx2 -O3' 'FC=ifx'
                          'FCFLAGS=-march=core-avx2 -O3'
                          '--enable-mpi-ext=affinity,pcollreq'
                          '--enable-mpi-fortran=all'
                          '--enable-mpi1-compatibility' '--enable-static'
                          '--with-cma'
                          '--with-libevent=/data/ashterenli/pe-hbv4/1.0/install/libevent-2.1.12-stable'
                          '--with-hwloc=/data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/hwloc-2.12.2'
                          '--with-ucx=/data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/ucx-1.20.0'
                          '--with-zlib=/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1'
                          '--without-cuda'
                          '--with-pmix=/data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/pmix-5.0.9'
                          '--with-prrte=/data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/prrte-3.0.12'
                          '--with-slurm' '--without-xpmem'
                          '--prefix=/data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/openmpi-5.0.9'
                Built by: anton.shterenlikht
                Built on: Mon Feb 16 13:33:49 UTC 2026
              Built host: tst5-hbv4-1
              C bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the ifx compiler and/or Open MPI,
                          does not support the following: array subsections,
                          direct passthru (where possible) to underlying Open
                          MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: no
  Wrapper compiler rpath: runpath
              C compiler: icx
     C compiler absolute: /data/ashterenli/pe-hbv4/1.0/install/intel-2025.3.1.55/compiler/2025.3/bin/icx
  C compiler family name: CLANG
      C compiler version: 21.1.1 (icx 2025.3.2.20260112)
            C++ compiler: icpx
   C++ compiler absolute: /data/ashterenli/pe-hbv4/1.0/install/intel-2025.3.1.55/compiler/2025.3/bin/icpx
           Fort compiler: ifx
       Fort compiler abs: /data/ashterenli/pe-hbv4/1.0/install/intel-2025.3.1.55/compiler/2025.3/bin/ifx
         Fort ignore TKR: yes (!DEC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, Event lib: yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
          MPI extensions: affinity
 Fault Tolerance support: yes
          FT MPI support: yes
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
         MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.9)
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.9)
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.9)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                 MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                 MCA btl: uct (MCA v2.1.0, API v3.3.0, Component v5.0.9)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.9)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.9)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.9)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v5.0.9)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.9)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                MCA smsc: xpmem (MCA v2.1.0, API v1.0.0, Component v5.0.9)
             MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.9)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                 MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
                          v5.0.9)
                MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.9)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.9)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
                          v5.0.9)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                  MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.9)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v5.0.9)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.9)
                MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.9)
                 MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
                          v5.0.9)
                 MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.9)
                 MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.9)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.9)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.9)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v5.0.9)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v5.0.9)
$

[ashterenli]

3. "And what is the file system that you are using to run your tests on?"

I usually build in /dev/shm, and hence run the tests there too.
I now tried setting HDF5_PARAPREFIX to locally attached ssd, or lustre - same apparent hang

4. "How many processes are you using to run t_mpi?"

just 4:

$ export HDF5_PARAPREFIX=/data/ashterenli
$ mpirun -n 4 ./t_mpi
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=tst5-hbv4-1
MPI-process 1. hostname=tst5-hbv4-1
MPI-process 3. hostname=tst5-hbv4-1
MPI-process 2. hostname=tst5-hbv4-1
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type

<hang>

5. "I would probably start by excluding as many optional components as possible, e.g. for not using ucc or hcoll by setting --mca coll ^ucc,hcoll"

no difference:

$ mpirun  --mca coll ^ucc,hcoll -n 4 ./t_mpi
===================================
MPI functionality tests
===================================
--------------------------------
Proc 0: *** MPIO 1 write Many read test...
--------------------------------
MPI-process 0. hostname=tst5-hbv4-2
MPI-process 3. hostname=tst5-hbv4-2
MPI-process 1. hostname=tst5-hbv4-2
MPI-process 2. hostname=tst5-hbv4-2
--------------------------------
Proc 0: *** MPIO File size range test...
--------------------------------
MPI_Offset is signed 8 bytes integral type

Also, this is how I configure hdf5 with intel:

        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.14.6
                  Configured on: Mon Feb 16 15:38:20 UTC 2026
                  Configured by: anton.shterenlikht@tst5-hbv4-1
                    Host system: x86_64-pc-linux-gnu
              Uname information: Linux tst5-hbv4-1 6.6.96.2-2.azl3 #1 SMP PREEMPT_DYNAMIC Thu Sep 11 02:15:17 UTC 2025 x86_64 x86_64 x86_64
GNU/Linux
                       Byte sex: little-endian
             Installation point: /data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/u_openmpi/5.0.9/hdf5-1.14.6

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:  -L/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1/lib
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/openmpi-5.0.9/bin/cc
                       CPPFLAGS:
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API -I/localscratch/1137025235/1771256251/hdf
5-1.14.6/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1/include
                        C Flags: -std=c11 -Wbad-function-cast -O2 -fPIC -march=core-avx2
                     H5 C Flags:   -std=c99 -Wall   -Wl,-s   -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /data/ashterenli/pe-hbv4/1.0/install/u_intel/2025.3.1.55/openmpi-5.0.9/bin/ftn
                  Fortran Flags: -O2 -fPIC -march=core-avx2
               H5 Fortran Flags:  -free -warn all     -O3
               AM Fortran Flags:
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: no

                           Java: no


Features:
---------
                     Parallel HDF5: yes
  Parallel Filtered Dataset Writes: yes
                Large Parallel I/O: yes
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib)
                  _Float16 support: no
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no

and with gcc (just -march is different):

        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
                   HDF5 Version: 1.14.6
                  Configured on: Mon Feb 16 15:38:15 UTC 2026
                  Configured by: anton.shterenlikht@tst5-hbv4-2
                    Host system: x86_64-pc-linux-gnu
              Uname information: Linux tst5-hbv4-2 6.6.96.2-2.azl3 #1 SMP PREEMPT_DYNAMIC Thu Sep 11 02:15:17 UTC 2025 x86_64 x86_64 x86_64
GNU/Linux
                       Byte sex: little-endian
             Installation point: /data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/u_openmpi/5.0.9/hdf5-1.14.6

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:  -L/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1/lib
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/openmpi-5.0.9/bin/cc
                       CPPFLAGS:
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API -I/localscratch/1137025235/1771256260/hdf
5-1.14.6/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/data/ashterenli/pe-hbv4/1.0/install/zlib-1.3.1/include
                        C Flags: -Wbad-function-cast -O2 -fPIC -march=znver4
                     H5 C Flags:  -std=c99  -Wall -Wcast-qual -Wconversion -Wextra -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmiss
ing-include-dirs -Wshadow -Wundef -Wwrite-strings -pedantic -Wno-c++-compat -Wlarger-than=2560 -Wlogical-op -Wframe-larger-than=16384 -Wpack
ed-bitfield-compat -Wsync-nand -Wno-unsuffixed-float-constants -Wdouble-promotion -Wtrampolines -Wstack-usage=8192 -Wmaybe-uninitialized -Wd
ate-time -Warray-bounds=2 -Wc99-c11-compat -Wduplicated-cond -Whsa -Wnormalized -Wnull-dereference -Wunused-const-variable -Walloca -Walloc-
zero -Wduplicated-branches -Wformat-overflow=2 -Wformat-truncation=1 -Wattribute-alias -Wshift-overflow=2 -Wattribute-alias=2 -Wmissing-prof
ile -Wc11-c2x-compat -fstdarg-opt -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wbad-function-cast -Wcast-align -Wformat -Wimplicit-f
unction-declaration -Wint-to-pointer-cast -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpoi
nter-sign -Wpointer-to-int-cast -Wredundant-decls -Wstrict-prototypes -Wswitch -Wunused-but-set-variable -Wunused-variable -Wunused-function
 -Wunused-parameter -Wincompatible-pointer-types -Wint-conversion -Wshadow -Wrestrict -Wcast-function-type -Wmaybe-uninitialized -Wcast-alig
n=strict -Wno-aggregate-return -Wno-inline -Wno-missing-format-attribute -Wno-missing-noreturn -Wno-overlength-strings -Wno-jump-misses-init
 -Wstrict-overflow=2 -Wno-suggest-attribute=const -Wno-suggest-attribute=noreturn -Wno-suggest-attribute=pure -Wno-suggest-attribute=format
-Wno-suggest-attribute=cold -Wno-suggest-attribute=malloc -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /data/ashterenli/pe-hbv4/1.0/install/u_gcc/15.2.0/openmpi-5.0.9/bin/ftn
                  Fortran Flags: -O2 -fPIC -march=znver4
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow
-pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type
 -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-t
emporaries -O3
               AM Fortran Flags:
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: no

                           Java: no


Features:
---------
                     Parallel HDF5: yes
  Parallel Filtered Dataset Writes: yes
                Large Parallel I/O: yes
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib)
                  _Float16 support: yes
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no