Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement functionality for measures search. #957

Open
OkonSamuel opened this issue Jul 19, 2022 · 2 comments
Open

implement functionality for measures search. #957

OkonSamuel opened this issue Jul 19, 2022 · 2 comments

Comments

@OkonSamuel
Copy link
Member

Is your feature request related to a problem? Please describe.
It would be great to have built into MLJ a way for searching measures suitable to a given predictions yhat and target y, similar to the model searching functionality already implemented.

Describe the solution you'd like
So I propose something along these lines, measures(matching(yhat, y)), which would return a list of measures that can be used suitable for predictions yhat and target y.

@ablaom
Copy link
Member

ablaom commented Jul 25, 2022

Okay this design won't do because of ambiguity with models(matching(X, y)). Also I don't like the fact that matching is forcing us to couple model search code with measure search code. Measures will eventually be independent of the whole stack, except StatisticalTraits.jl.

  1. One suggestion is to deprecate predicates as arguments of models and do something like this:
  • models() acts as currently
  • models(str) acts as currently, for str a string or regex
  • models((X, y)) acts like models(matching(X, y)) does now.
  • models(p) falls back to assume p is a predicate (such as matching(X, y)) but this behaviour is deprecated, as not needed (we have filter for that).
  • measures((yhat, y)) finds all measures compatible with (yhat, y)
  1. A breaking alternative is to modify the third item above and allow models(X, y) to behave as models(matching(X, y)) does now, but that would disallow the possibility that X is a string, say, and we would have to disallow predicates immediately.

@ablaom
Copy link
Member

ablaom commented Aug 23, 2022

Another nice enhancement would be to allow kwargs for searching over trait values, as in models(package_name="ScikitLearn", is_supervised=true). Currently we have the ugly

models() do m
    m.package_name(m) == ScikitLearn" &&
    m.is_supervised
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants