Open MPI program stops to work due to ptrace linux kernel vulnerability CVE-2026-46333

[jfikar]

System: updated Rocky Linux 9.8, kernel 5.14.0-687.24.1.el9_8.x86_64, I expect the same problem on all RHEL 9 based distributions

Ptrace vul. CVE-2026-46333 patched:

cat /proc/sys/kernel/yama/ptrace_scope
3

Open MPI: from repo ver 5.0.5

Problem:

I've old binary of Lammps (https://github.com/lammps/lammps) compiled with Open MPI. It for sure worked before. Then the patched prtace kernel was installed and the calculations do not run.

Command to run 20 instances on a single machine (SMT) as a normal user:
mpirun -n 20 --mca btl sm,self lmp-4Jul2026 -in in.difus

Dmesg fills with:

[ 4994.830355] __report_access: 1084759 callbacks suppressed
[ 4994.830361] ptrace attach of "lmp-4Jul2026 -in in.difus"[8347] was attempted by "lmp-4Jul2026 -in in.difus"[8346]
[ 4994.830364] ptrace attach of "lmp-4Jul2026 -in in.difus"[8352] was attempted by "lmp-4Jul2026 -in in.difus"[8351]
[ 4994.830364] ptrace attach of "lmp-4Jul2026 -in in.difus"[8339] was attempted by "lmp-4Jul2026 -in in.difus"[8338]
[ 4994.830377] ptrace attach of "lmp-4Jul2026 -in in.difus"[8348] was attempted by "lmp-4Jul2026 -in in.difus"[8352]
[ 4994.830378] ptrace attach of "lmp-4Jul2026 -in in.difus"[8338] was attempted by "lmp-4Jul2026 -in in.difus"[8342]
[ 4994.830378] ptrace attach of "lmp-4Jul2026 -in in.difus"[8342] was attempted by "lmp-4Jul2026 -in in.difus"[8341]
[ 4994.830386] ptrace attach of "lmp-4Jul2026 -in in.difus"[8334] was attempted by "lmp-4Jul2026 -in in.difus"[8333]
[ 4994.830392] ptrace attach of "lmp-4Jul2026 -in in.difus"[8344] was attempted by "lmp-4Jul2026 -in in.difus"[8343]
[ 4994.830392] ptrace attach of "lmp-4Jul2026 -in in.difus"[8343] was attempted by "lmp-4Jul2026 -in in.difus"[8347]
[ 4994.830394] ptrace attach of "lmp-4Jul2026 -in in.difus"[8352] was attempted by "lmp-4Jul2026 -in in.difus"[8348]
[ 4999.831469] __report_access: 1086171 callbacks suppressed
[ 4999.831474] ptrace attach of "lmp-4Jul2026 -in in.difus"[8347] was attempted by "lmp-4Jul2026 -in in.difus"[8346]
[ 4999.831479] ptrace attach of "lmp-4Jul2026 -in in.difus"[8343] was attempted by "lmp-4Jul2026 -in in.difus"[8347]
[ 4999.831479] ptrace attach of "lmp-4Jul2026 -in in.difus"[8344] was attempted by "lmp-4Jul2026 -in in.difus"[8343]
[ 4999.831480] ptrace attach of "lmp-4Jul2026 -in in.difus"[8339] was attempted by "lmp-4Jul2026 -in in.difus"[8338]
[ 4999.831482] ptrace attach of "lmp-4Jul2026 -in in.difus"[8349] was attempted by "lmp-4Jul2026 -in in.difus"[8348]
[ 4999.831486] ptrace attach of "lmp-4Jul2026 -in in.difus"[8348] was attempted by "lmp-4Jul2026 -in in.difus"[8352]
[ 4999.831486] ptrace attach of "lmp-4Jul2026 -in in.difus"[8352] was attempted by "lmp-4Jul2026 -in in.difus"[8351]
[ 4999.831487] ptrace attach of "lmp-4Jul2026 -in in.difus"[8351] was attempted by "lmp-4Jul2026 -in in.difus"[8350]
[ 4999.831487] ptrace attach of "lmp-4Jul2026 -in in.difus"[8350] was attempted by "lmp-4Jul2026 -in in.difus"[8349]
[ 4999.831493] ptrace attach of "lmp-4Jul2026 -in in.difus"[8336] was attempted by "lmp-4Jul2026 -in in.difus"[8335]
[ 5004.832165] __report_access: 1089327 callbacks suppressed
[ 5004.832170] ptrace attach of "lmp-4Jul2026 -in in.difus"[8343] was attempted by "lmp-4Jul2026 -in in.difus"[8347]
[ 5004.832180] ptrace attach of "lmp-4Jul2026 -in in.difus"[8347] was attempted by "lmp-4Jul2026 -in in.difus"[8346]
[ 5004.832183] ptrace attach of "lmp-4Jul2026 -in in.difus"[8344] was attempted by "lmp-4Jul2026 -in in.difus"[8345]
[ 5004.832184] ptrace attach of "lmp-4Jul2026 -in in.difus"[8349] was attempted by "lmp-4Jul2026 -in in.difus"[8350]
[ 5004.832190] ptrace attach of "lmp-4Jul2026 -in in.difus"[8348] was attempted by "lmp-4Jul2026 -in in.difus"[8349]
[ 5004.832191] ptrace attach of "lmp-4Jul2026 -in in.difus"[8338] was attempted by "lmp-4Jul2026 -in in.difus"[8339]
[ 5004.832194] ptrace attach of "lmp-4Jul2026 -in in.difus"[8333] was attempted by "lmp-4Jul2026 -in in.difus"[8334]
[ 5004.832198] ptrace attach of "lmp-4Jul2026 -in in.difus"[8340] was attempted by "lmp-4Jul2026 -in in.difus"[8341]
[ 5004.832198] ptrace attach of "lmp-4Jul2026 -in in.difus"[8352] was attempted by "lmp-4Jul2026 -in in.difus"[8348]
[ 5004.832198] ptrace attach of "lmp-4Jul2026 -in in.difus"[8335] was attempted by "lmp-4Jul2026 -in in.difus"[8336]
[ 5009.833514] __report_access: 1090243 callbacks suppressed
[ 5009.833520] ptrace attach of "lmp-4Jul2026 -in in.difus"[8339] was attempted by "lmp-4Jul2026 -in in.difus"[8338]
[ 5009.833530] ptrace attach of "lmp-4Jul2026 -in in.difus"[8334] was attempted by "lmp-4Jul2026 -in in.difus"[8333]
[ 5009.833531] ptrace attach of "lmp-4Jul2026 -in in.difus"[8344] was attempted by "lmp-4Jul2026 -in in.difus"[8343]
[ 5009.833532] ptrace attach of "lmp-4Jul2026 -in in.difus"[8352] was attempted by "lmp-4Jul2026 -in in.difus"[8351]
[ 5009.833539] ptrace attach of "lmp-4Jul2026 -in in.difus"[8342] was attempted by "lmp-4Jul2026 -in in.difus"[8341]
[ 5009.833540] ptrace attach of "lmp-4Jul2026 -in in.difus"[8338] was attempted by "lmp-4Jul2026 -in in.difus"[8342]
[ 5009.833548] ptrace attach of "lmp-4Jul2026 -in in.difus"[8337] was attempted by "lmp-4Jul2026 -in in.difus"[8336]
[ 5009.833548] ptrace attach of "lmp-4Jul2026 -in in.difus"[8333] was attempted by "lmp-4Jul2026 -in in.difus"[8337]
[ 5009.833554] ptrace attach of "lmp-4Jul2026 -in in.difus"[8351] was attempted by "lmp-4Jul2026 -in in.difus"[8350]
[ 5009.833554] ptrace attach of "lmp-4Jul2026 -in in.difus"[8350] was attempted by "lmp-4Jul2026 -in in.difus"[8349]

The calculation looks like running just fine, but stops in the middle. The ssh connections are dropped and new ssh connections stalled. I've spotted also an OOM Killer. Not sure, what exactly the problem is. It might be simply too many messages in the logs.

When compiled with mpich/3.4.3-ofi, everything works as expected with just:
mpirun -n 20 lmp-4Jul2026 -in in.difus

Is there a way to avoid pthrace in Open MPI?

EDIT: it is a memory leak. After about 13 minutes from start the Open MPI version takes about 100GB RAM, while the mpich version stays at 2GB max RAM all the time.

[hppritcha]

Could you try running LAMMPS with

mpirun --mca smsc ^cma -n 20 lmp-4Jul2026 -in in.difus

[jfikar]

That seems to work, thanks! However, I'm not able to get rid of the --mca btl sm,self part.

This works: pirun -n 20 --mca smsc ^cma --mca btl sm,self lmp-4Jul2026 -in in.difus1

Without it I'm getting:

mpirun -n 20 --mca smsc ^cma lmp-4Jul2026 -in in.difus
--------------------------------------------------------------------------
No components were able to be opened in the btl framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      miranda2
  Framework: btl
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another.  This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used.  Your MPI job will now abort.

You may wish to try to narrow down the problem;

 * Check the output of ompi_info to see which BTL/MTL plugins are
   available.
 * Run your application with MPI_THREAD_SINGLE.
 * Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
   if using MTL-based communications) to see exactly which
   communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_mpi_instance_init failed
  --> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
[miranda2:00000] *** An error occurred in MPI_Init
[miranda2:00000] *** reported by process [2176843777,0]
[miranda2:00000] *** on a NULL communicator
[miranda2:00000] *** Unknown error
[miranda2:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[miranda2:00000] ***    and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node miranda2 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------