Varying feature set

[plvaudry]

It seems implicit that, because each input is represented by a dict, the feature set could vary over time. Some features could sometimes be missing. Some features could appear or disappear over time. Also, I noticed that some feature selection methods have been implemented, so there is yet another way the feature set could vary.

However, I have not found documentation on how implemented models behave in such circumstances.

If the behaviour varies from model to model, what are the differences?

[smastelini]

Hi @plvaudry, that's a really good question. I think there's not a definitive answer for all the cases. Ideally, all the learners, transformers, and so on should be robust against such changes. Luckily, the majority of them already are. @MaxHalford wrote a nice set of tests to check these properties.

I do know that k-NN models and ensemble.SRPClassifier are yet to become capable of dealing with emerging/missing features. That is in the TODO lists of @jacobmontiel and me.

I am more knowledgeable of the tree module. In this case, you can find the information about how they deal with missing and emerging features in the documentation (currently, only in the development version).

I think @MaxHalford, @jacobmontiel, @raphaelsty, @gbolmier, @VaysseRobin, @AdilZouitine, and @hoanganhngo610 can give you more hints about other modules.

I believe documenting these details is a task worth putting in River's roadmap. :)

[Aaryanverma]

@smastelini Suppose a new feature is added into the test set which was not present in the train set. Can you explain theoretically how a classification model is going to make a new branch to accommodate the new feature? On which basis model decides whether the new feature will positively or negatively affect the prediction?

[smastelini]

In this case, it will not. Models are prepared to deal with new categories in an already existing feature. Trees, for instance, will deal with this situation differently while learning or predicting.

In the case of a new feature that only appears during testing, this info will be simply ignored by the models. This is the case with decision trees and I think it will pretty much be the same with other learners.

Now, if after testing these instances are passed to the learners, then the additional feature it's going to be leveraged by the models. Trees can use the new feature to create more splits (if the feature is informative).

[smastelini]

@MaxHalford, would you like to add anything regarding other families of learners?

[Aaryanverma]

@smastelini Thanks for clarifying, if possible I would like to know the same explanation about linear models as well.

[raphaelsty]

When a model in the river.linear_model module encounters an unknown feature, the model will initialize a new weight for this feature. River provides different methods to initialize the weights (zeros, normal, constant). You can find them in the river.optim.initializer module. @smastelini @MaxHalford could give more information about how missing features are handled.

Few words for the Naive Bayes module with a single example naive_bayes.BernoulliNB computes P(features | class) and P(class) to make a prediction. When an unknown feature is encountered, the probability P(class) takes precedence over P(features | class) which is set to (0 + alpha) / (count(class) + alpha * 2). alpha is a customizable class attribute which is set to 1 by default.

naive_bayes.MultinomialNB and naive_bayes.ComplementNB are a bit different but unknown / missing features are handled by design in these models also.