Is it possible to have incremental task labels from 'dataset_benchmark'?

[thipokKub]

I created a composite dataset using multiple train_stream and test_stream. However the task results did not change (it got stuck at task 0). This cause the problem down the line. From the official docs there are mentions of task_labels which presence in ni_benchmark and nc_benchmark which return instance of NCScenario and NIScenario respectively. But both of them seems to mess with internal dataset structure e.g. shuffling data, and specifying n_experiences. So is there an official way to create incremental task label as current_experience?

Noted that this problem came after the code from #785, and following Official Tutorial

mix_scenario = dataset_benchmark(
    new_train_stream, new_test_stream
)

print(mix_scenario.train_stream[1].dataset[0][-1]) # print 0, expected 1

P.S. right now, I am using experiment.current_experience as suppose to task

[vlomonaco]

Hi @thipokKub! Thanks for taking the time to report your issue :)
This seems more of a usability question rather then a bug, so I'm moving this in the discussions section.

As for your question, "avalanche_dataset" is meant to be used with custom Datasets definitions. In the official tutorial this example is reported:

# Alternatively, task labels can also be a list (or tensor)
# containing the task label of each pattern

train_MNIST_task0 = AvalancheDataset(train_cifar10, task_labels=0)
test_MNIST_task0 = AvalancheDataset(test_cifar10, task_labels=0)

train_cifar10_task1 = AvalancheDataset(train_cifar10, task_labels=1)
test_cifar10_task1 = AvalancheDataset(test_cifar10, task_labels=1)

scenario_custom_task_labels = dataset_benchmark(
    [train_MNIST_task0, train_cifar10_task1],
    [test_MNIST_task0, test_cifar10_task1]
)

print('Without custom task labels:',
      generic_scenario.train_stream[1].task_label)

print('With custom task labels:',
      scenario_custom_task_labels.train_stream[1].task_label)

As you can see above you can specify the task_label of each experience when you create the corresponding AvalancheDataset.

[Mattdl]

Great question and answer, I was looking to solve exactly the same issue (:
Could be useful to have a reference to this in the dataset_benchmark documentation.

[AndreaCossu]

We could also think about making this a short how-to, similar to the one in the notebook for dataloaders and buffers.

[vlomonaco]

Yeah, I think @lrzpellegrini has already in mind to do an How-to notebook on the "AvalancheDataset" and connected methods! :)
as for the dataset_benchmark doc.. this is just an alias of "create_multi_dataset_generic_benchmark" which has the doc. We will fix this ASAP. We should also add an example for each method in the API doc... as Pytorch does!

[MagicHealer]

@vlomonaco I am confused with the example code given above. Is it right for this line to use Cifar10 instead of mnist?
train_MNIST_task0 = AvalancheDataset(train_cifar10, task_labels=0)

[vlomonaco]

you're right @MagicHealer it is not :)
It should be train_mnist and test_mnist!