Problem with River-ML regressors.

[SirPopiel]

Hello,
I have been experimenting with River-ML for online regression under concept/covariate drift.
In particular, I have experimented with multiple regressors, such as ARF and SRP.
However, I have some issues, as prequential RMSE goes up dramatically (e.g. in the billions) after a while. This, with the same methods implemented in Scikit-multiflow, does not happen. (For ARF: 0.55 rmse with scikit-multiflow, while in river it gets to billions).
This also happens on different, synthetic data I am using, though not always.
Additionally, it only happens with adaptive random forest, SRP (with linear models (even regularized)) and online linear models, while for EWA (with linear models) or KNN, this does not happen.
I would be grateful if you could point me to possible issues, as I am also fine using scikit-multiflow, but, at the same time, it sports less options (for instance, it does not have SRPRegression), so I would like to have River methods work fine.
Thank you in advance.
Filippo

PS

This is the sample code in River (in sk-multiflow it is really similar, but the prequential rmse is computed manually, as it does not seem to support prequential RMSE. Anyways, I checked the problem is not in evaluate progressive score by also using the same procedure for river.).

model=ensemble.AdaptiveRandomForestRegressor()
metric = metrics.RMSE()
dataset = stream.iter_pandas(X = df.iloc[:,:-2], y = df.iloc[:,-2] )
% here, it pre-trains on the data i actually have for training
for x, y in itertools.islice(dataset, train_size):
    model.predict_one(x)
    model.learn_one(x, y)
final = evaluate.progressive_val_score(dataset = dataset, model = model, metric = metric, print_every=100).get()

[MaxHalford]

Hello! Would it be possible to share your dataset with us? You can do so privately.

It's going to be hard/impossible for us to reproduce your issue without it.

[SirPopiel]

Okay, thank you! I still have a big difference in performances (1.1 rmse river vs 0.55 rmse skmultiflow), but I guess that is due to some other changes in the base hyperparameters and the implementation. Anyways, thank you very much for the quick support!

[smastelini]

Hi @SirPopiel. I intend to investigate that. When setting the leaf prediction strategy to "mean" in both tools, do you perceive such differences?

The dataset you sent us has around 9k samples, which could be considered small for streaming applications. Thus, the performance differences could be related to the leaf prediction models (for instance, the regularization type and/or the applied learning rate). But if the differences persist when using the target mean at the leaves, then there might be something odd going on.

[smastelini]

@SirPopiel, your observation is correct. A few things have changed during the merger (default hyperparameter values, code simplification and so on). In skmultiflow the leaves' linear models were hardcoded. In River, you can specifically set up the model you want.

For instance, by using:

    model = preprocessing.StandardScaler() | ensemble.AdaptiveRandomForestRegressor(
        split_confidence=0.1, leaf_prediction="adaptive", seed=42,
        leaf_model=linear_model.LinearRegression(
            optimizer=optim.RMSProp(), l2=0.01
        )
    )

I got 0.83 of RMSE. I also changed the split_confidence value to match the one in skmultiflow.

[smastelini]

You can tweak your model to reduce the error even further. Other feature standardization procedures are also available in the preprocessing module.

[SirPopiel]

@smastelini @MaxHalford Thank you for your answers. I don't need to optimize its performance, but it is nice to have understood the reason for the behaviour! I'll tell you if i encounter any other problem :)