Interestingly, changed:

• Prevent mpi4py from calling Finalize (not sure if this had any effect because mpi4py might not call it if MPI was initialized before loading it...)
• Ensure MPI_Init and MPI_Finalize are called correctly in the unit test (though the error begs to differ)
• Ensure static construction order of GIL and DataProvider

and this left me with the error:

Abort(805361423) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
PMPI_Init(109): Cannot call MPI_INIT or MPI_INIT_THREAD more than once
LLVMSymbolizer: error reading file: No such file or directory

when running with MPI enabled. MPI disabled gives me the same segfault, but the above errors don't actually segfault and the unit test program exits normally...

After talking with @hellkite500 and @PhilMiller, I can confirm at the very least that this is due to an atexit hook created in Python (my guess is via esmpy) that uses MPI functions.

By changing the unit test tear down function to:

static void TearDownTestSuite()
{
    gil_->getModule("atexit").attr("_run_exitfuncs")();
    #if NGEN_WITH_MPI
    MPI_Finalized(&mpi_final);
    if (mpi_final == 0) {
        MPI_Finalize();
    }
    #endif
}

where I forcibly call all atexit functions created in the Python interpreter, the program exits safely, with no errors or segfaults.

Working with Justin, we validated using the PMPI interfaces to 'detour' the MPI_Finalize call in ESMF.

The way this works is that we provide a 'profiling' implementation of MPI_Finalize in a small shared object. This implementation will actually be a no-op. ESMF's call to MPI_Finalize will resolve to call this implementation, and hence have no harmful side effects. Then, in our code (the tests and NGen.cpp), we explicitly call PMPI_Finalize as implemented by the MPI library, to actually shut down MPI.

As an update for this, esmf-org/esmf#234 was merged which solves half of this issue (assuming the user has an ESMF build with this feature). The other half left is ensuring destruction of ESMF Grid/Mesh/Field objects used in the Forcings Engine python code (NOAA-OWP/ngen-forcing#14).