Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compute_metrics for metrics with different signatures #74

Open
sarahmish opened this issue Aug 25, 2021 · 0 comments
Open

compute_metrics for metrics with different signatures #74

sarahmish opened this issue Aug 25, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@sarahmish
Copy link

Environment Details

Please indicate the following details about the environment in which you found the bug:

  • SDMetrics version: 0.3.2
  • Python version: 3.7

Error Description

When using sdmetrics.compute_metrics we get None for metrics that were not able to resolve keyword arguments. In most cases, we expect the user to pass a dictionary for metrics to compute_metrics but if the signature of these metrics differ, we get an error that causes us to catch it then store None in its location.

For example, take the following two time series metrics TSFCDetection and TSFClassifierEfficacy. The first one is expected to be called with

TSFCDetection.compute(data, sampled, entity_columns=entity_columns)

and the second one with an additional argument target

TSFClassifierEfficacy.compute(data, sampled, entity_columns=entity_columns, target=target)

compute_metrics will pass target to both classes which TSFCDetection cannot handle, thus crashing.

Steps to reproduce

import pandas as pd
from sdmetrics.timeseries import TSFCDetection, TSFClassifierEfficacy

length = 10

real = pd.DataFrame({
    "seq_index": [1, 2, 3] * length,
    "dim_0": [0, 0, 0] * length,
    "dim_1": [4, 4, 4] * length
})

synth = pd.DataFrame({
    "seq_index": [1, 2, 3] * length,
    "dim_0": [1, 0, 0] * length,
    "dim_1": [4, 4, 3] * length
})

metrics = {
    'TSFClassifierEfficacy': TSFClassifierEfficacy,
    'TSFCDetection': TSFCDetection
}

sdmetrics.compute_metrics(metrics, real, synth, entity_columns=['seq_index'], target='dim_0')
@sarahmish sarahmish added the bug Something isn't working label Aug 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant