Merge pull request #3445 from easybuilders/4.9.x

release EasyBuild v4.9.3

.github/workflows/linting.yml

@@ -22,7 +22,8 @@ jobs:
     - name: install Python packages
       run: |
         pip install --upgrade pip
-        pip install --upgrade flake8
+        # fix to this version for develop branch (to avoid needing to fix geant4.py)
+        pip install --upgrade "flake8"
 
     - name: Run flake8 to verify PEP8-compliance of Python code
       run: flake8

RELEASE_NOTES

@@ -6,6 +6,63 @@ These release notes can also be consulted at http://easybuild.readthedocs.org/en
 The latest version of easybuild-easyblocks provides 259 software-specific easyblocks and 43 generic easyblocks.
 
 
+v4.9.3 (14 September 2024)
+--------------------------
+
+update/bugfix release
+- minor updates, including:
+  - update custom easyblock for Tensorflow for versions 2.14 + 2.15 (#3303)
+  - add support for versions >= 2024a to MCR easyblock (#3369)
+  - update custom easyblock for Bazel for versions >= 7.0 (#3370)
+  - don't consider lib/release for impi >= 2021.11 (#3375)
+  - update ORCA easyblock for version 6 (#3395)
+  - update custom easyblock for psmpi to support the renaming of a couple of options in 5.10.0-1 (#3420)
+  - update sanity check in OpenFOAM easyblock, since there's no `modifyMesh` in OpenFOAM 12 (#3434)
+  - fix sanity check for Geant4 >= v11.2 (#3439)
+- minor enhancements, including:
+  - enhance psmpi easyblock to activate CUDA support when it is a dependency + make check for static libs in MPICH easyblock optional (#2787)
+  - make a project environment and manage LOAD_PATH for JuliaPackage (#3239)
+  - set build type for PyTorch explicitely (#3332)
+  - add `sanity_check_test_inputs` custom easyconfig parameter, mapping for zen4, and support building of non-stable versions to LAMMPS easyblock (#3336)
+  - add RISCV64 support and update config options of new versions of Extrae (#3339)
+  - set `$ESMFMKFILE` environment variable in environment module for ESMF (#3368)
+  - use build dir for big files/folders while building TensorFlow (#3371)
+  - update `PythonPackage` easyblock to allow installation of Python packages with `$PIP_REQUIRE_VIRTUALENV` set + move temporary pip folder into build dir (#3374)
+  - add RISC-V support to Boost easyblock (#3376)
+  - add support for generating `.gem` files from `.gemspec` files and support for `preinstallopts` in the `RubyGem` easyblock (#3381)
+  - simplify install step in custom easyblock for `Tkinter` (#3382)
+  - add MSA and specially PMIx support for ParaStationMPI (#3383)
+  - update custom easyblock for PETSc to consider `include/suitesparse` subdirectory for SuiteSparse headers (#3391)
+  - enhance custom easyblock for GCC to use `with-arch` option for nvptx with 13.1+ (#3396)
+  - support revisions in crates extraction of cargo packages (#3405)
+  - set `$R_LIBS_USER` in `RPackage` easyblock to avoid picking up on R packages installed in home directory (#3407)
+  - update custom easyblock for QuantumESPRESSO to be aware of MPI tests being disabled in EasyBuild configuration (#3412)
+  - disable the version check of pip in Python packages by defining `$PIP_DISABLE_PIP_VERSION_CHECK` (#3427)
+  - make sure user packages are not used in sanity check of PythonBundle (#3435)
+  - enhance OpenFOAM easyblock to also build the plugins for OpenFOAM >= v2406 (#3436)
+  - enhance binutils easyblock to explicitely pass msgpack configure option (#3438)
+- various bug fixes, including:
+  - patch `ctypes` for Python installations when filtering `LD_LIBRARY_PATH` + fix path to `ldconfig` when using alternate sysroot (#3352)
+  - also consider 'normalised' package name with underscore rather than dash in EasyBuild easyblock (#3358)
+  - use PRRTE MCA environment variable for oversubscription in OpenMPI easyblock (#3360)
+  - fix import of `parse` in `openssl_wrapper.py` for Python 2.7 (#3364)
+  - make sure that OpenFOAM's wmake can find MPFR and GMP if CGAL >= 5.0 (header-only) is used (#3366)
+  - remove dummy license server info from ANSYS (#3377)
+  - explicitly call `csh` in custom easyblock for WPS (#3384)
+  - use relative paths to object files when compiling shared libraries in the OpenFOAM easyblock (#3388)
+  - change extra `$PATH` entry for OCaml to use `opam/default` rather than `opam/system` (#3390)
+  - force `--without-unwind` for ARM and add `--with-libz` option for all architectures in custom easyblock for Extrae (#3393)
+  - fix `--sanity-check-only` for impi (#3403)
+  - fix crash in Cargo easyblock when no crates are specified (#3404)
+  - fix double initialization of `Cargo` by `CargoPythonPackage` by removing incorrect custom `__init__` implementation + fix use of `super()` in PALM easyblock (since that doesn't work with Python 2.7) (#3406)
+  - fix typo in log message in `PythonPackage` easyblock (#3408)
+  - make sure to raise an error if `pick_python_cmd` returns `False` for Python bundles/packages (#3430)
+  - don't wipe build environment before building opam in install step of OCaml easyblock (#3443)
+- other changes:
+  - merge of the `ConfigureMake` and `CMakeMake` versions of the easyblock for QuantumESPRESSO (#3338)
+  - remove redundant backslashes in GEANT easyblock (#3394)
+
+
 v4.9.2 (12 June 2024)
 ---------------------
 

easybuild/easyblocks/__init__.py

@@ -43,7 +43,7 @@
 # recent setuptools versions will *TRANSFORM* something like 'X.Y.Zdev' into 'X.Y.Z.dev0', with a warning like
 #   UserWarning: Normalizing '2.4.0dev' to '2.4.0.dev0'
 # This causes problems further up the dependency chain...
-VERSION = LooseVersion('4.9.2')
+VERSION = LooseVersion('4.9.3')
 UNKNOWN = 'UNKNOWN'
 
 

easybuild/easyblocks/a/ansys.py

@@ -49,16 +49,16 @@ def __init__(self, *args, **kwargs):
 
     def install_step(self):
         """Custom install procedure for ANSYS."""
-        licserv = self.cfg['license_server']
-        if licserv is None:
-            licserv = os.getenv('EB_ANSYS_LICENSE_SERVER', 'license.example.com')
-        licport = self.cfg['license_server_port']
-        if licport is None:
-            licport = os.getenv('EB_ANSYS_LICENSE_SERVER_PORT', '2325:1055')
-
         # Sources (e.g. iso files) may drop the execute permissions
         adjust_permissions('INSTALL', stat.S_IXUSR)
-        cmd = "./INSTALL -silent -install_dir %s -licserverinfo %s:%s" % (self.installdir, licport, licserv)
+        cmd = "./INSTALL -silent -install_dir %s" % self.installdir
+        # E.g. license.example.com or license1.example.com,license2.example.com
+        licserv = self.cfg.get('license_server', os.getenv('EB_ANSYS_LICENSE_SERVER'))
+        # E.g. '2325:1055' or just ':' to use those defaults
+        licport = self.cfg.get('license_server_port', os.getenv('EB_ANSYS_LICENSE_SERVER_PORT'))
+        if licserv is not None and licport is not None:
+            cmd += ' -licserverinfo %s:%s' % (licport, licserv)
+
         run_cmd(cmd, log_all=True, simple=True)
 
         adjust_permissions(self.installdir, stat.S_IWOTH, add=False)

easybuild/easyblocks/b/bazel.py

@@ -164,7 +164,11 @@ def configure_step(self):
         # We want to enforce it using the JDK we provided via modules
         # This is required for Power where Bazel does not have a JDK, but requires it for building itself
         # See https://github.com/bazelbuild/bazel/issues/10377
-        bazel_args += ' --host_javabase=@local_jdk//:jdk'
+        if LooseVersion(self.version) >= LooseVersion('7.0'):
+            # Option changed in Bazel 7.x, see https://github.com/bazelbuild/bazel/issues/22789
+            bazel_args += ' --tool_java_runtime_version=local_jdk'
+        else:
+            bazel_args += ' --host_javabase=@local_jdk//:jdk'
 
         # Link C++ libs statically, see https://github.com/bazelbuild/bazel/issues/4137
         static = self.cfg['static']

easybuild/easyblocks/b/binutils.py

@@ -148,6 +148,15 @@ def configure_step(self):
                 else:
                     libs.append(libz_path)
 
+        msgpackroot = get_software_root('msgpack-c')
+        if LooseVersion(self.version) >= LooseVersion('2.39'):
+            if msgpackroot:
+                self.cfg.update('configopts', '--with-msgpack')
+            else:
+                self.cfg.update('configopts', '--without-msgpack')
+        elif msgpackroot:
+            raise EasyBuildError('msgpack is only supported since binutils 2.39. Remove the dependency!')
+
         env.setvar('LIBS', ' '.join(libs))
 
         # explicitly configure binutils to use / as sysroot

easybuild/easyblocks/b/boost.py

@@ -53,7 +53,7 @@
 from easybuild.tools.filetools import apply_regex_substitutions, read_file, symlink, which, write_file
 from easybuild.tools.modules import get_software_root, get_software_version
 from easybuild.tools.run import run_cmd
-from easybuild.tools.systemtools import AARCH64, POWER, UNKNOWN
+from easybuild.tools.systemtools import AARCH64, POWER, RISCV64, UNKNOWN
 from easybuild.tools.systemtools import get_cpu_architecture, get_glibc_version, get_shared_lib_ext
 
 
@@ -326,6 +326,8 @@ def sanity_check_step(self):
                     lib_mt_suffix += '-a64'
                 elif get_cpu_architecture() == POWER:
                     lib_mt_suffix += '-p64'
+                elif get_cpu_architecture() == RISCV64:
+                    lib_mt_suffix += '-r64'
                 else:
                     lib_mt_suffix += '-x64'
 

easybuild/easyblocks/e/easybuildmeta.py

@@ -133,7 +133,10 @@ def install_step(self):
         try:
             subdirs = os.listdir(self.builddir)
             for pkg in self.easybuild_pkgs:
-                seldirs = [x for x in subdirs if x.startswith(pkg)]
+                # also consider "normalized" package name, with dashes ('-') replaced by underscores ('_'),
+                # which is being enforced by recent versions of setuptools (>= 69.0.3?)
+                pkg_norm = pkg.replace('-', '_')
+                seldirs = [x for x in subdirs if x.startswith(pkg) or x.startswith(pkg_norm)]
                 if len(seldirs) != 1:
                     # setuptools is optional since it may be available in the OS;
                     # vsc-install and vsc-base sources are optional,

easybuild/easyblocks/e/esmf.py

@@ -152,6 +152,11 @@ def make_module_extra(self):
         """Add install path to PYTHONPATH or EBPYTHONPREFIXES"""
         txt = super(EB_ESMF, self).make_module_extra()
 
+        # set environment variable ESMFMKFILE
+        # see section 9.9 in https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/node10.html
+        esmf_mkfile_path = os.path.join(self.installdir, "lib", "esmf.mk")
+        txt += self.module_generator.set_environment('ESMFMKFILE', esmf_mkfile_path)
+
         if self.cfg['multi_deps'] and 'Python' in self.cfg['multi_deps']:
             txt += self.module_generator.prepend_paths('EBPYTHONPREFIXES', '')
         else:
@@ -173,7 +178,7 @@ def sanity_check_step(self):
             binaries = ['ESMF_PrintInfo', 'ESMF_PrintInfoC', 'ESMF_Regrid', 'ESMF_RegridWeightGen',
                         'ESMF_Scrip2Unstruct', 'ESMF_WebServController']
 
-        libs = ['libesmf.a', 'libesmf.%s' % get_shared_lib_ext()]
+        libs = ['esmf.mk', 'libesmf.a', 'libesmf.%s' % get_shared_lib_ext()]
         custom_paths = {
             'files': [os.path.join('bin', x) for x in binaries] + [os.path.join('lib', x) for x in libs],
             'dirs': ['include', 'mod'],

easybuild/easyblocks/e/extrae.py

@@ -30,6 +30,9 @@
 
 from easybuild.easyblocks.generic.configuremake import ConfigureMake
 from easybuild.tools.modules import get_software_root
+from easybuild.tools import LooseVersion
+from easybuild.tools.systemtools import RISCV64
+from easybuild.tools.systemtools import get_cpu_architecture
 
 
 class EB_Extrae(ConfigureMake):
@@ -42,14 +45,27 @@ def configure_step(self):
         self.cfg.update('configopts', "--with-mpi=%s" % get_software_root(self.toolchain.MPI_MODULE_NAME[0]))
 
         # Optional dependences
-        deps = {
-            'binutils': ('', '--with-binutils=%s', ''),
-            'Boost': ('', '--with-boost=%s', ''),
-            'libdwarf': ('', '--with-dwarf=%s', '--without-dwarf'),
-            'libunwind': ('', '--with-unwind=%s', ''),
-            'libxml2': (' --enable-xml --enable-merge-in-trace', '', ''),
-            'PAPI': ('--enable-sampling', '--with-papi=%s', '--without-papi'),
-        }
+        # Both --enable-xml and --with-dwarf options are no longer available from 4.1.0 version
+        # Instead, --with-xml is used
+        if LooseVersion(self.version) >= LooseVersion('4.1.0'):
+            deps = {
+                'binutils': ('', '--with-binutils=%s', ''),
+                'Boost': ('', '--with-boost=%s', ''),
+                'libunwind': ('', '--with-unwind=%s', '--without-unwind'),
+                'libxml2': ('--enable-merge-in-trace', '--with-xml=%s', ''),
+                'PAPI': ('--enable-sampling', '--with-papi=%s', '--without-papi'),
+                'zlib': ('', '--with-libz=%s', ''),
+            }
+        else:
+            deps = {
+                'binutils': ('', '--with-binutils=%s', ''),
+                'Boost': ('', '--with-boost=%s', ''),
+                'libdwarf': ('', '--with-dwarf=%s', '--without-dwarf'),
+                'libunwind': ('', '--with-unwind=%s', '--without-unwind'),
+                'libxml2': (' --enable-xml --enable-merge-in-trace', '', ''),
+                'PAPI': ('--enable-sampling', '--with-papi=%s', '--without-papi'),
+            }
+
         for (dep_name, (with_opts, with_root_opt, without_opt)) in deps.items():
             dep_root = get_software_root(dep_name)
             if dep_root:
@@ -64,6 +80,10 @@ def configure_step(self):
         # TODO: make this optional dependencies
         self.cfg.update('configopts', "--without-dyninst")
 
+        # Needed to build in RISC-V architectures
+        if get_cpu_architecture() == RISCV64:
+            self.cfg.update('configopts', "--enable-posix-clock")
+
         super(EB_Extrae, self).configure_step()
 
     def sanity_check_step(self):

easybuild/easyblocks/g/gcc.py

@@ -366,6 +366,70 @@ def prep_extra_src_dirs(self, stage, target_prefix=None):
             'versions': versions
         }
 
+    def map_nvptx_capability(self):
+        """
+        Convert PTX ISA architecture passed via EasyBuild configs to a version which is understood by GCC.
+        Valid architecture strings include 'sm_30', 'sm_35', 'sm_53', 'sm_70', 'sm_75', 'sm_80'.
+        (as of GCC 14.1.0). Some additional architectures may be mapped.
+        See: https://github.com/gcc-mirror/gcc/commit/de0ef04419e90eacf0d1ddb265552a1b08c18d4b
+
+        As this list is updated regularly, try to parse the GCC source file (gcc/config/nvptx/nvptx.opt)
+        and extract the supported architectures and their mapping. Based on the result, determine the lowest
+        architecture required to support all 'cuda_compute_capabilities' and return this value.
+        """
+        cuda_cc_list = build_option('cuda_compute_capabilities') or self.cfg['cuda_compute_capabilities']
+        architecture_mappings_file = os.path.join(self.cfg['start_dir'], 'gcc', 'config', 'nvptx', 'nvptx.opt')
+        architecture_mappings_flag = "march-map="
+        architecture_mappings_replacement = "misa=,"
+
+        # Determine which compute capabilities are configured. If there are none, return immediately.
+        if cuda_cc_list is None:
+            return None
+        cuda_sm_list = [f"sm_{cc.replace('.', '')}" for cc in cuda_cc_list]
+
+        if not os.path.exists(architecture_mappings_file):
+            warn_msg = f"Tried to parse nvptx.opt but file {architecture_mappings_file} was not found. " \
+                       "Please check the path and update the EasyBlock if necessary!"
+            self.log.warning(warn_msg)
+            return None
+
+        # We want to read the mappings found in the GCC sources and create a map for this.
+        # We're searching for the following pattern:
+        # march-map=sm_32
+        # Target RejectNegative Alias(misa=,sm_30)
+        gcc_architecture_mappings = {}
+        file_content = read_file(architecture_mappings_file).splitlines()
+        for line_idx, line in enumerate(file_content):
+            line = line.strip()
+            if line.startswith(architecture_mappings_flag):
+                key = line.split('=')[1]
+                # Mapped architecture can be found in the following line
+                line = file_content[line_idx + 1]
+                if architecture_mappings_replacement not in line:
+                    warn_msg = "Tried to parse nvptx.opt but failed to extract mapped architectures! " \
+                                f"Expected to find substring '{architecture_mappings_replacement}' in " \
+                                f"line {line_idx + 1} but found '{line}'. Choosing default of GCC."
+                    self.log.warning(warn_msg)
+                    # Bail out, since results of mapping cannot be trusted
+                    return None
+                value = line.split(architecture_mappings_replacement)[1].rstrip(')')
+                gcc_architecture_mappings[key] = value
+        self.log.info(f"Available architecture mappings in GCC {self.version}:\n {str(gcc_architecture_mappings)}")
+
+        # Map compute capabilities to GCC ones
+        # If no compute capability can be mapped, stick to default of GCC and return None
+        gcc_cc = [gcc_architecture_mappings[cc] if cc in gcc_architecture_mappings else None for cc in cuda_sm_list]
+        self.log.info(f"Mapped architectures: {str(cuda_sm_list)} -> {str(gcc_cc)}")
+        if any(cc is None for cc in gcc_cc):
+            self.log.info("At least one architecture could not be mapped. Choosing default of GCC.")
+            return None
+
+        # Get the lowest architecture mapping and return it
+        sorted_gcc_cc = sorted(gcc_cc)
+        self.log.info("Choosing first architecture in sorted list as default nvptx "
+                      f"architecture: {str(sorted_gcc_cc)}")
+        return sorted_gcc_cc[0]
+
     def run_configure_cmd(self, cmd):
         """
         Run a configure command, with some extra checking (e.g. for unrecognized options).
@@ -544,6 +608,10 @@ def configure_step(self):
                         self.create_dir("build-nvptx-gcc")
                         target = 'nvptx-none'
                         self.cfg.update('configopts', "--enable-newlib-io-long-long")
+                        if LooseVersion(self.version) >= LooseVersion('13.1.0'):
+                            cuda_cc = self.map_nvptx_capability()
+                            if cuda_cc:
+                                self.cfg.update('configopts', f'--with-arch={cuda_cc}')
                     else:
                         # compile AMD GCN target compiler
                         self.create_dir("build-amdgcn-gcc")