Create script for automated regression testing

[ilectra]

Fixes #8 .
This is stil WIP, I need to

• check that all files are copied properly to the build directory and paths are correct
• possibly convert to pytest

But you can already use it from inside the directory that the test sits in build, after copying the file with the reference values over.
@edbennett are the values we're checking the right ones? Or do we need the palette value to a gazillion decimals?

[qiUip]

In terms of functionality, this works great! A few suggestions in the review.

I also noticed that the results change based on compilation options, e.g. when compiling for sp2n (--enable-Sp --enable-Nc=4) and running Test_hmc_IwasakiGauge I got different results than the same test when compiled for su3 (--enable-Nc=3). I'm not sure these values will make a lot of sense without a very strict compile and run parameters.

[edbennett]

Something I should have considered earlier: since this uses a Metropolis algorithm, checking the final value of the plaquette is only valid if the version being compared against accepts the change. (In the case of a rejected change, then the value of the plaquette output is the input value, and only depends on the code we are touching if it changes in such a way that the change is accepted instead, which is unlikely.)

Provided we consistently stick with the same test case (and have verified that it accepts), then I think this is fine, but we should be careful about generalising this.

[ilectra]

• To add more parameters/references to check for: nthreads, CPUvsGPU
• read references from command line (if possible) I don't think this is a good idea. It should be easy to add a line in the ...._expected.txt file.

[ilectra]

To be able to tell if the run was with GPU or CPU, look for nvidia::memalloc or unified memory in the output.

[ilectra]

I think this is ready for a final review. Please have a look and add test cases values that you find useful to the Test_hmc_Sp_WilsonFundFermionGauge_expected.txt file. @qiUip @edbennett @asifsamiarain

[ilectra]

Those two expected values files have to be adjusted for the extra test parameters and line numbers. @qiUip

[ilectra]

Add usage note to script

[edbennett]

As you noted, a note on usage would be useful. You might want to run ruff format or similar to tidy up some very minor things. Other than that, looks good to me!

[ilectra]

The lat.checkoint file checksum is machine dependent, therefore not appropriate for regression tests (unless we keep note of the machine where the reference values were generated as well). A better value to use for reference is LINK_TRACE, found in line 9 of the lat.checkpoint header.
Do we still want to keep the checksum for more detailed comparisons? In this case, the user would have to generate the checksum before the changes in the code they want to check for exact bit-wise identity, and then compare it with the one after. A command line option for the pytest, eg. --checksum, could take no argument to just report the checksum after the run, or take the value to compare with as an argument. Either way, it'll be a 2-step process, where the user will have to run pytest with the old version of the code to generate the checksum value, then re-compile with the new version of the code and run pytest again to do the comparison. Does this seem worth the effort @edbennett ?

[edbennett]

Having a tool that makes it easier to do a checksum comparison before and after a change, so you can verify that a trajectory is bitwise compatible when run on the same machine, would be useful. Whether it's worth the effort of course depends on how much effort it would be. I'd say it's worth a single-digit number of hours of effort, but not that much more than that.