Figure of Merit?

[vsoch]

Hi! I saw that your README says "Coming soon" for the FOM. Is there something to calculate or derive from the output? Thanks!

[vladotomov]

Hi @vsoch,

We have FOM reporting and initial GPU capability in the branch gpu-opt. We will merge this soon.

Here are some sample runs:
2D cpu:
lrun -n 8 remhos -dim 2 -epm 1024 -o 3 -p 14 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cpu
2D gpu:
lrun -n 8 remhos -dim 2 -epm 1024 -o 3 -p 14 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cuda
3D cpu:
lrun -n 8 remhos -dim 3 -epm 512 -o 2 -p 10 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cpu
3D gpu:
lrun -n 8 remhos -dim 3 -epm 512 -o 2 -p 10 -dt -1.0 -tf 0.5 -ho 3 -lo 5 -fct 2 -vs 1 -ms 5 -no-vis -pa -d cuda

The main performance-related options are the device (-d), number of tasks (-n), the elements per task (-epm), and the finite element order (-o). Appropriate mesh and partitioning are generated automatically. The product of (-n) and (-epm) determines the mesh size. The order (-o) can also be increased, resulting in more work per mesh element. For example, for weak scaling, vary (-n) and fix (-epm); for strong scaling, make sure the product of (-n)*(-epm) is constant.

We might do further changes / optimizations in the branch.

[vsoch]

This is great! We definitely need more GPU benchmarks. I'll add this one to my list, and if you want to keep the issue open, am happy to report back here (likely close to the weekend, I'm currently on travel).

[vsoch]

@vladotomov it looks like there isn't a spack variant for GPU:

https://packages.spack.io/package.html?name=remhos

I can build the dependencies separately. Aside from having cuda, do you know the flags / configure options that are needed for the build to support GPU? If you have a development build with GPU somewhere I can go from that too. Thank you!

[vladotomov]

Once MFEM is built for GPU, the build of Remhos follows. Same as Laghos.

Let me know if that doesn't work.

[tzanio]

Aside from git tags and version numbers, the Spack packages for Laghos and Remhos should be very similar, if not identical

[vsoch]

Build was a success! I'll test it soon.

[vsoch]

Actually I take it back - I was using the main branch. When I do gpu-opt there is an issue with hypre:

7.156 /opt/mfem/linalg/sparsemat.hpp:774:16: note:   candidate expects 4 arguments, 0 provided
7.156 /opt/mfem/linalg/sparsemat.hpp:777:16: note: candidate: 'mfem::SparseMatrix* mfem::Add(Array<SparseMatrix*>&)'
7.156   777 | SparseMatrix * Add(Array<SparseMatrix *> & Ai);
7.156       |                ^~~
7.156 /opt/mfem/linalg/sparsemat.hpp:777:16: note:   candidate expects 1 argument, 0 provided
7.156 /opt/mfem/linalg/sparsemat.hpp:780:6: note: candidate: 'void mfem::Add(const SparseMatrix&, real_t, DenseMatrix&)'
7.156   780 | void Add(const SparseMatrix &A, real_t alpha, DenseMatrix &B);
7.156       |      ^~~
7.156 /opt/mfem/linalg/sparsemat.hpp:780:6: note:   candidate expects 3 arguments, 0 provided
7.156 /opt/mfem/linalg/hypre.hpp:977:17: note: candidate: 'mfem::HypreParMatrix* mfem::Add(real_t, const HypreParMatrix&, real_t, const HypreParMatrix&)'
7.156   977 | HypreParMatrix *Add(real_t alpha, const HypreParMatrix &A,
7.156       |                 ^~~
7.156 /opt/mfem/linalg/hypre.hpp:977:17: note:   candidate expects 4 arguments, 0 provided
7.820 make: *** [makefile:135: remhos.o] Error 1
------
Dockerfile:33
--------------------
  32 |     WORKDIR /opt
  33 | >>> RUN git clone -b gpu-opt https://github.com/CEED/Remhos /opt/remhos && \
  34 | >>>     cd /opt/remhos && \
  35 | >>>     make
  36 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c git clone -b gpu-opt https://github.com/CEED/Remhos /opt/remhos &&     cd /opt/remhos &&     make" did not complete successfully: exit code: 2

[vsoch]

I was using a pinned (now older) version - I'll try updating it.

[vsoch]

OK, working again with version updates! Sorry for the noise. Will again test soon.