Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review GMLogLikelihood metric test noise #56

Open
csala opened this issue Mar 24, 2021 · 0 comments
Open

Review GMLogLikelihood metric test noise #56

csala opened this issue Mar 24, 2021 · 0 comments
Labels
documentation Docs, user guides or API internal The issue doesn't change the API or functionality

Comments

@csala
Copy link
Contributor

csala commented Mar 24, 2021

The GMLogLikelihood metric was added to cover the metrics that existed in the original SDGym iteration, which used GM Log Likelhood metric over datasets that were simulated using GaussianMixtures.

However, even though the implementation was optimized and improved to make the output as stable and meaningful as possible, the scores produced when this metric is run on datasets which not simulated from GMs tends to be very noisy and may produce inconsistent results between runs. As a consequence of this, the ranking-based integration test fails randomly.

We may want to remove this metric from the ranking test and have a separated one which is tested using GM simulated data, and also add a disclaimer in the documentation indicating what this metric is best suited for.

@npatki npatki added internal The issue doesn't change the API or functionality documentation Docs, user guides or API labels Jul 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Docs, user guides or API internal The issue doesn't change the API or functionality
Projects
None yet
Development

No branches or pull requests

2 participants