Suggestions/Improvements while approaching Avalanche

[francesco-p]

Hello everyone,
I'm trying to approach Avalanche to run some experiments. I have made some considerations that might help the development.

1. Why not introducing some (real) simple baselines experiments as "template" (I know there is reproducible CL repo, but it is focused on reproduce research instead of spreading how to use Avalanche)

2. In the documentation it might be helpful to introduce an atomic use case for each class or function you defined (like in Pytorch Documentation). For example for the nc_benchmark documentation:

from avalanche.benchmarks.datasets import CIFAR100
import torchvision.transforms as T
from avalanche.benchmarks import nc_benchmark

# Benchmark = Dataset + Generator

# Dataset
train_cifar = CIFAR100('../data', train=True, download=True, transform=T.ToTensor())
test_cifar = CIFAR100('../data', train=False, download=True, transform=T.ToTensor())

# Generator
scenario = nc_benchmark(train_cifar, test_cifar, n_experiences=10, seed=1337,task_labels=False)

for experience in scenario.train_stream:
    print("Start of experience: ", experience.current_experience)
    print("Current Classes: ", experience.classes_in_this_experience)
    
    # Train...

I might be greedy but when I approach a new library I want a quick simple example. If you can do it for each module you can do "lego programming" as in Pytorch.

3. Why not wrapping timm as model provider for Avalanche? It should be easy to do so coz all models are split in backbone + fc... I personally believe it is the SOTA of network architecture, therefore it could be a very appealing feature.

[vlomonaco]

Hi @francesco-p! Thanks for your feedback! :)

Why not introducing some (real) simple baselines experiments as "template" (I know there is reproducible CL repo, but it is focused on reproduce research instead of spreading how to use Avalanche)

Agreed. I think the examples directory is already providing some kind of support in this sense. Are there any specific features you see are missing in this sense or do we just need to improve the examples made available there?

In the documentation it might be helpful to introduce an atomic use case for each class or function you defined (like in Pytorch Documentation).

This is already on our priority list! Very much needed!

Why not wrapping timm as model provider for Avalanche?

We already have pytorchcv as an (optional) dependency in Avalanche. Still, I think you can already use timm as it is already compatible with Avalanche. What features are missing in your opinion in its support?

[AntonioCarta]

Thanks for the feedback, these are all important points.

Why not introducing some (real) simple baselines experiments as "template" (I know there is reproducible CL repo, but it is focused on reproduce research instead of spreading how to use Avalanche)

Can you explain why the reproducible-cl is not adequate for this use case? We have some other examples, but I think starting from a reproducible baseline is better in general, because you have all the hyperparameters already setup according to the best literature results.

In the documentation it might be helpful to introduce an atomic use case for each class or function you defined (like in Pytorch Documentation). For example for the nc_benchmark documentation:

I totally agree with this. We need to start adding more examples. You are not greedy, everyone does it ;)
I will open an issue about this. btw, this is a useful contribution if you want to help.

Why not wrapping timm as model provider for Avalanche? It should be easy to do so coz all models are split in backbone + fc... I personally believe it is the SOTA of network architecture, therefore it could be a very appealing feature.

I agree that Avalanche should be compatible with other libraries in the pytorch ecosystem. However, I don't see much value in a thin wrapper. like Vincenzo said, we already wrap pytorchcv. The problem is that there are tens of useful libraries, everyone uses different libraries, and we can't possibly provide thin wrappers around all of them. For example, another library that can be easily integrated with Avalanche is torchmetrics.

Another related problems with thin wrappers is that the package would end up with a lot of dependencies, which is not ideal.

[francesco-p]

(1). Yeah it is super nice to have scripts to replicate sota methods, but I was thinking more to some "compositional use cases" i.e. better span over avalanche potential. For example, I'm used to FACIL and, while much less powerful, it is very straightforward to use it and the output is very clear (taw, tag matrices).

(3). If you want to maintain sota standards then I guess timm is the right choice, because it introduces latest models and weights (which is crucial, we all know the struggle of training beasts). For example they introduce all ViTs variations (small, base, large), along with pretrained weights, they also introduced the last fb thingy https://arxiv.org/abs/2201.03545 .

(2). Yes of course I will like to contribute, indeed I'm writing some small examples (and I just spot this small error in docs :D)

[AntonioCarta]

We are definitely open to specific suggestions for more "compositional use cases". We have examples, how-to, and the from-zero-to-hero which cover a lot of details and best practices, but there are still gaps in the documentation.

I don't think we will ever be as intuitive as FACIL because Avalanche tries to be much more comprehensive, both in terms of scenarios and strategies. This necessarily adds a bit of complexity, which we can hide, but not always.