-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create script for automated regression testing #13
base: develop
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In terms of functionality, this works great! A few suggestions in the review.
I also noticed that the results change based on compilation options, e.g. when compiling for sp2n (--enable-Sp --enable-Nc=4
) and running Test_hmc_IwasakiGauge I got different results than the same test when compiled for su3 (--enable-Nc=3
). I'm not sure these values will make a lot of sense without a very strict compile and run parameters.
Something I should have considered earlier: since this uses a Metropolis algorithm, checking the final value of the plaquette is only valid if the version being compared against accepts the change. (In the case of a rejected change, then the value of the plaquette output is the input value, and only depends on the code we are touching if it changes in such a way that the change is accepted instead, which is unlikely.) Provided we consistently stick with the same test case (and have verified that it accepts), then I think this is fine, but we should be careful about generalising this. |
Co-authored-by: Mashy Green <[email protected]>
|
SOme improvements in error checking and reporting.
Future test reference value files added, will have to be linked over explicitly here as well. autotools does not do wildcards.
To be able to tell if the run was with GPU or CPU, look for |
I think this is ready for a final review. Please have a look and add test cases values that you find useful to the |
@@ -0,0 +1 @@ | |||
8.8.8.8 1.1.1.1 0.269298793 633bf471 3a22ad20 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those two expected values files have to be adjusted for the extra test parameters and line numbers. @qiUip
Add usage note to script |
As you noted, a note on usage would be useful. You might want to run |
Add some more test parameters and expected values - in progress... Modify saveInterval in hmc tests to produce outputs. Fix number of threads to 1 if not specified.
The |
Having a tool that makes it easier to do a checksum comparison before and after a change, so you can verify that a trajectory is bitwise compatible when run on the same machine, would be useful. Whether it's worth the effort of course depends on how much effort it would be. I'd say it's worth a single-digit number of hours of effort, but not that much more than that. |
Fixes #8 .
This is stil WIP, I need tobuild
directory and paths are correctpytest
But you can already use it from inside the directory that the test sits in
build
,after copying the file with the reference values over.@edbennett are the values we're checking the right ones? Or do we need the palette value to a gazillion decimals?