Skip to content

v0.4.0

Latest
Latest
Compare
Choose a tag to compare
@lsawade lsawade released this 26 Jun 12:38
· 13 commits to main since this release
v0.4.0
8c3a12d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 TL;DR

Physics updates:

  • Support for elastic SH wave simulation (forward simulation support)
  • Support for Poro-elastic domains (forward simulation support)
  • Support for Cosserat domains (forward simulation support)

Performance Improvements:

  • Memory layout optimization. Solver time on par with SPECFEM2D.

Code maintainance:

  • New macros for better code generation
  • Introduced iterator framework to define parallelism over different solver loops.

Refactoring

  • Split examples into examples and benchmarks directories.

Configuration

  • Introduces CMAKE presets for regularly used build configurations

Maintenance

  • Updates Kokkos version to 4.6.1.
  • Adds support for AMD GPUs
  • Adds support for parallel testing

📋 What's New

Support for elastic SH wave

Wave propagation of an SH wave in an elastic medium in response to a transverse force source Ricker source time function. The domain has one horizontal interface in the vertical center of the domain splitting a slower upper from a faster lower layer.

SH wave simulation with interface

Support for Poro-elastic domains

Below wave propagation simulation in a homogeneous poroelastic medium in response to a medium central downward force in acting on.

Poro-elastic Simulation in a homogeneous medium

Support for Cosserat domains

Below a simulation of wave propagation in a Cosserat medium in response to a central downward force. On the left the norm of the displacement wavefield and on the right the rotational field around the transverse axis.

Wave propagation in a homogeneous Cosserat medium

New macros for better code generation

This feature streamlines the codebase by eliminating redundant code, making it easier to maintain. In addition to improved maintainability, this feature significantly reduces the amount of code required to implement a new medium. Below the old and new way to introduce a new data container to the solver, by the example of acoustic-isotropic media.

The old way of defining the property_container

For each medium/property combination, the previous code required the user to define the following properties container, that is used to store the properties for relevant elements.

template <>
struct properties_container<acoustic isotropic specialization> {

  using ViewType = typename Kokkos::View<type_real ***, Kokkos::LayoutLeft,
                                         Kokkos::DefaultExecutionSpace>;

  int nspec; ///< total number of acoustic spectral elements
  int ngllz; ///< number of quadrature points in z dimension
  int ngllx; ///< number of quadrature points in x dimension
  ViewType rho_inverse;
  ViewType::HostMirror h_rho_inverse;
  ViewType kappa;
  ViewType::HostMirror h_kappa;

  properties_container(const int nspec, const int ngllz, const int ngllx)
      : nspec(nspec), ngllz(ngllz), ngllx(ngllx),
        rho_inverse("specfem::compute::properties::rho_inverse", nspec, ngllz,
                    ngllx),
        h_rho_inverse(Kokkos::create_mirror_view(rho_inverse)),
        kappa("specfem::compute::properties::kappa", nspec, ngllz, ngllx),
        h_kappa(Kokkos::create_mirror_view(kappa)) {}

	template<...>
  KOKKOS_FORCEINLINE_FUNCTION void
  load_device_properties(index, point_property) const { ... }

  template<...>
  load_device_properties(index, point_property) const { ... }

  template<...> 
  inline void
  load_host_properties(index, point_property) const { ... }

  template<...>
  inline void
  load_host_properties(index, point_property) const { ... }

  void copy_to_device() {
    Kokkos::deep_copy(rho_inverse, h_rho_inverse);
    Kokkos::deep_copy(kappa, h_kappa);
  }

  void copy_to_host() {
    Kokkos::deep_copy(h_rho_inverse, rho_inverse);
    Kokkos::deep_copy(h_kappa, kappa);
  }

  template<...>
  inline void assign(index, point_property) const { ... }
};

Even after omitting quite a few code blocks with ... for clarity, leaving ~25% of the code, it is clear that this is not very user friendly, nor maintainable, and too specialized with the perspective of implementing new physics, which is why we needed a more concise, simpler way to approach new data.

The new way of defining the property_container → data_container

Using macros we can wrap all of the above medium/property specific code into a single line that takes in the variable names and generates all of the above required code during the preprocessing stage.

template <acoustic isotropic specialization>
struct data_container {
  constexpr static auto dimension = specfem::dimension::type::dim2;
  constexpr static auto medium_tag = MediumTag;
  constexpr static auto property_tag =
      specfem::element::property_tag::isotropic;

  DATA_CONTAINER(kappa, mu, rho)
};

A complete documentation for available macros can be found here

CMake Presets

Prior to this release the one always need to manually define the all flags. For example, for a CUDA release build, the user needed to

cmake3 -S . -B build/release-cuda -D CMAKE_BUILD_TYPE=Release -D BUILD_TESTS=ON -D Kokkos_ENABLE_CUDA=ON -D Kokkos_ARCH_<architecture>=ON
cmake --build build/release-cuda

to build the software. We added CMakePresets.json for a variety of “standard” compilations. Which allow for building using

cmake --preset release-cuda
cmake --build --preset release-cuda

We want to advertise this feature in particular since cmake allows for CMakeUserPresets.json which if located in the SPECFEM++ root directory takes precedence over CMakePresets.json and can be used for super custom user installation without having to populate 4 lines of the command line every time you want to build. The list of “standard” presets are:

"release"          - Default Release -- SIMD enabled
"release-nosimd"   - Release -- SIMD disabled
"release-cuda"     - Release -- CUDA enabled
"release-hip"      - Release -- HIP enabled
"release-frontier" - Release Frontier -- HIP enabled
"debug"            - Default Debug -- SIMD enabled
"debug-cuda"       - Debug -- CUDA enabled
"debug-nosimd"     - Debug -- SIMD disabled

Performance

The plots below show the performance of SPECFEM++ compared to SPECFEM2D:

Elastic simulation

Performance on CPU -- Homogeneous Elastic Performance on GPU -- Homogeneous Elastic

Acoustic Simulation

Performance on CPU -- Homogeneous Acoustic Performance on GPU -- Homogeneous Acoustic

Elastic-Acoustic coupled simulation

Performance on CPU -- Elastic-Acoustic Performance on GPU -- Elastic-Acoustic

New Contributors

Statistics

Commits and PRs since the last Release

Changelog

👨‍🔬 New physics

🚀 Performance enhancement

✚ Enhancement

  • Issue 472 -- Implement templated array reading for multidimensional fortran arrays. Reading ibool, coordinates, partials by @lsawade in #475
  • Add function to access field of a medium by @icui in #459
  • Issue 484/498/499/500 -- Implements reading of ispec_is_, materials, and mass matrices. by @lsawade in #501
  • Issue 502 - Implements new read_array templates dim3 absorbing boundaries write/read by @lsawade in #503
  • Issue 504 - Implement the reading of coupled interfaces. by @lsawade in #505
  • Issue 508 -- Implement final bout of structus to read in the 3D mesh by @lsawade in #519
  • Updates compute kernels and properties to use macros, updates writers, updates parameter parser by @Rohit-Kakodkar in #540
  • Implements computation kernels [Part 1] by @Rohit-Kakodkar in #560
  • [MAIN PR] Read the 3D Mesh (Issue #390) by @lsawade in #476
  • Computational Kernels implementation [Part 2] by @Rohit-Kakodkar in #570
  • Implements macros for operator[] function in mesh materials by @Rohit-Kakodkar in #621
  • Create shared parent class for sh and p-sv. by @icui in #596
  • Check timescheme updates all DOFs and elements by @Rohit-Kakodkar in #668
  • Implement register arrays using standard library by @Rohit-Kakodkar in #756
  • Implemented max reduction for compute properties / kernels by @Rohit-Kakodkar in #754
  • Issue 704, 705, 706 - new macros for code block, variable declaration and template instantiation by @icui in #762
  • Issue 783 - Add option to write wavefield every N timesteps by @icui in #787
  • Issue 780 - Update to new macro for MEDIUM_TAGS and ELEMENT_TYPES by @icui in #782
  • Issue 793 - Write wavefield data of different time steps into a single file. by @icui in #796
  • Issue 797 - Implements compile-time conditional execution of "extra physics" [poroelastic damping] by @lsawade in #798
  • Updated chunk vector view layouts by @Rohit-Kakodkar in #764
  • Issue 790 - Add string utilities to relax configuration string by @icui in #803
  • Added tiling for Range loops by @Rohit-Kakodkar in #817
  • Fixes writing of the wavefield with multiple media and adds a Python plotting script. by @lsawade in #827
  • Added elastic dim3 element attributes by @lsawade in #838
  • Check for small jacobian by @Rohit-Kakodkar in #847
  • Reorder mass matrix computation by @Rohit-Kakodkar in #849
  • Issue 809 accessor by @icui in #841
  • Issue 812 - Implement point_partial_derivatives for dim3 by @icui in #831
  • Issue 809 du - implement Accessor for field derivatives by @icui in #852
  • Iterator and Index concepts by @Rohit-Kakodkar in #854
  • Issue 834 - Added small example directory by @lsawade in #862
  • Issue 876 - Implemented SIMD all of comparison, and a test suite testing the SIMD by @lsawade in #878
  • Issue 703 - HIP support by @lsawade in #872
  • Issue 905 - add is_close() utility and apply to point tests by @icui in #920
  • Issue 919 - Add tests for macros in point_data and update operator== by @icui in #921

📚 Documentation

  • Fix for readthedocs build failing by @int-ptr-ptr in #497
  • Issue 506 - Addressing comments on previous few commits by @lsawade in #510
  • Issue 580 - sphinx_design version number fix by @lsawade in #581
  • Issue 582 - update documentation for uv, boost, build/bin by @icui in #594
  • Issue 717 - Fix Kokkos links in documentataion. by @icui in #718
  • Issue 784 - Updated the parameter doc & added print function for assembly by @lsawade in #785
  • Issue 694 - Updated the Documentaion and removed Lambda G additional member docstring by @lsawade in #863
  • Issue 548 - Adds Solid-Solid SH example by @lsawade in #864
  • Resolves issues related to contribution documetation by @Rohit-Kakodkar in #908
  • 809 api doc by @lsawade in #913
  • Add poroelastic example. by @icui in #917
  • Issue 722 - Update documentation for cmake presets. by @icui in #922
  • CUBIT example by @Rohit-Kakodkar in #923
  • Add documentation on macros. by @icui in #930

Code Style Updates

  • Migrate from poetry to uv by @icui in #454
  • Issue 811 - Implement pitchfork style for point types. by @icui in #819

Fortran Meshfem2D/3D updates

Refactor

  • Issue 455 - Refactor IO/mesh/impl/fortran -> IO/mesh/impl/fortran/dim2 & create barebone specfem3d.cpp by @lsawade in #456
  • Issue 485 -- Refactor of mesh struct to split it in dim2 and dim3 folders by @lsawade in #486
  • Updated elastic tag to elastic_sv tag by @Rohit-Kakodkar in #494
  • Issue-489: refactor point_properties and property_container by @icui in #509
  • Issue-513: Merge some host and device access functions by @icui in #518
  • Issue-521 and 523: Rename medium::properties to material and merge material_properties and properties_container by @icui in #524
  • Refactor point_kernels and kernels_container by @icui in #549
  • Merge kernel and property writer by @icui in #551
  • [Main PR] Refactor point and container class of properties and kernels by @icui in #552
  • Issue 573 - Removes boost submodule by @lsawade in #575
  • Issue 629 - Update the medium_tag elastic_sv -> elastic_psv by @lsawade in #719
  • Issue 674 -- Updated the EM tag from electromagnetic_sv -> electromagnetic_te by @lsawade in #720
  • Issue 752 - Re-organize current macros by @icui in #753
  • Removed the seismogram type and replaced it with wavefield type by @lsawade in #799
  • Upated field derivatives and stress tensor layout by @Rohit-Kakodkar in #763
  • Issue 810 - Rename DimensionType, MediumType to DimensionTag, MediumTag by @icui in #820
  • Point properties/kernels to remove manual indexing by @Rohit-Kakodkar in #833
  • Issue 809 - Accessor for point_field, source, stress and boundary by @icui in #855
  • Issue 809 - Implements Point Index/Coordinates/Properties and associated unit tests by @lsawade in #839
  • Issue 702 material derivatives by @Rohit-Kakodkar in #887
  • Updates Kokkos iteration policies by @Rohit-Kakodkar in #902
  • Issue 809 - Kernels and Docs + doc fixes for Properties by @lsawade in #869
  • Add tests for partial_derivatives, sources, stress_intregrand, stress, & boundary. by @icui in #874
  • Issue 868 -- Made elastic isotropic point properties and kernels dimension templated. by @lsawade in #914

🐛 Bug Fixes

Configuration/CMake

  • Issue 398 - Implement dynamic testing by @lsawade in #464
  • Issue 463 -- Making header files configuration files by @lsawade in #467
  • Fix CMake preset for cuda by @icui in #664
  • Updated cmakelists to get git repo by @lsawade in #749
  • Implements new structure for examples (now benchmarks) executable directory (default ./bin) and fixes documentation by @lsawade in #825
  • Issue 856 - implemented a clean target by @lsawade in #894
  • Update homogeneous-cosserat benchmark stations by @maxlchien in #916

🔧 Maintenance

🧾 New tests

Full Changelog: v0.3.0...v0.4.0

Contributors

lsawade, Rohit-Kakodkar, and 5 other contributors
Assets 2
Loading