Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression tests discrepancies #55

Open
isaacaka opened this issue Nov 20, 2024 · 0 comments
Open

Regression tests discrepancies #55

isaacaka opened this issue Nov 20, 2024 · 0 comments

Comments

@isaacaka
Copy link
Member

isaacaka commented Nov 20, 2024

Regression tests discrepancies.

Running the failing cases using Matlab 2024 to reproduce the results.

The core issue seems to be the stochastic sampling nature of the code. For certain input files, the output seems to vary on each run and this is obviously problematic for a simple comparison based test framework.
The two failing tests are US-Syv* and US-ARM*.

  • Comparing our refactored matlab code generated reports to the provided reports for the different sites may not be possible in some cases.

Options:
For testing our matlab refactored code:

  • Pass the same bootstrap data to both refactored and non-refactored version matlab ustart cp (setting a seed may be sufficient in producing the same output reports)

For testing our python translation:

  • Pass the same bootstrap data to matlab ustar_cp and python ustar_cp and ensure that all functions produce the same numerical output to a few significant figures
  • Test that the distribution of samples which are sampled from the site data is similar when using python vs matlab

Also should note whether using Matlab 2024 vs 2018a is important for the testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant