Allow for multiple scoring metrics in RFECV

[ArturoSbr]

Workflow

In its current state, RFECV only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using k <= K features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run RFECV twice: one with scoring='precision' and another with scoring='recall'.
This is inefficient, as it implies running RFECV twice instead of once.

The cv_results_ attribute of GridSearchCV returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in RFECV would be extremely helpful.

Proposed solution

Notation

• K is the number of folds used for cross-validation.
• P is the total number of features available.
• p is the number of features tried at each step. That is, an integer such that min_features_to_select <= p <= P.
• m is one of M performance metrics passed by the user (e.g., 'precision').

Solution

User can pass a list of strings representing M predefined scoring metrics and at each step, the algorithm stores the performance metric of the k models trained with p <= P features.

The cv_results_ attribute of the resulting RFECV would now include the following keys for each metric m and fold k:

• 'split{k}_test_{m}'
• mean_test_{m}
• 'std_test_{m}'
• 'rank_test_{m}'

Example

rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)

Considerations

It is likely that rank_test_{m1} will differ from rank_test_{m2} for any pair of performance metrics m1 and m2. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

Describe alternatives you've considered, if relevant

Running RFECV as many times as there are metrics.

Additional context

I asked this question on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.

[bmreiniger]

For your "Considerations" section, I wonder if it makes sense to also adopt the refit approach from the hyperparameter tuners: pick one metric to choose the best k, or even define a custom strategy callable.

(Maybe this should keep in mind #21291 too?)

[ogrisel]

This sounds like a legitimate need that cannot be efficiently be addressed with the current API as far as I understand.

So we could indeed consider expanding RFECV to accept multiple metrics + passing a custom callable as done for GridSearchCV, see:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html

[glemaitre]

Did we improve a bit the situation by merging: #28360

[adrinjalali]

With #28360 the situation is much improved, but we need to be able to understand multiple metrics the same way as *SeachCV here. For now we assume a single value as metric value.

This is related to #28944 for me too, since we're choosing sth based on certain metrics values while understanding multiple metrics.