Train-Val Splitter for Experiences

[HamedHemati]

Hi everyone!

I have provided a simple function that splits an experience into training and validation parts:

    def split_exp_train_val(exp, train_ratio=0.9):
        classes = set(exp.dataset.targets)
        targets = torch.LongTensor(exp.dataset.targets)
        # Deep copy of the main experience for train and validation
        exp_train = deepcopy(exp)
        exp_val = deepcopy(exp)

        # Initialize indices
        ind_train = []
        ind_val = []

        # Add train and val indices per class
        for c in classes:
            ind_c = torch.where(targets == c)[0]
            ind_c = ind_c[torch.randperm(len(ind_c))]
            n_train_samples = int(train_ratio * len(ind_c))
            ind_train.append(ind_c[:n_train_samples])
            ind_val.append(ind_c[n_train_samples:])
        ind_train = torch.cat(ind_train)
        ind_val = torch.cat(ind_val)

        # Create subsets of the main dataset with the extracted indices
        exp_train.dataset = AvalancheSubset(exp_train.dataset, ind_train)
        exp_val.dataset = AvalancheSubset(exp_val.dataset, ind_val)
        
        return exp_train, exp_val

And it can be used as below:

    for experience in scenario.train_stream:
        train_exp, val_exp = split_exp_train_val(experience)
        cl_strategy.train(experiences=train_exp,
                                      eval_stream=val_exp)

What do you think about adding it to Avalanche Core? Is there something that I'm missing?

• This method can be easily extended to streams of experiences.

[AntonioCarta]

we already have this, which splits the entire stream.

[HamedHemati]

Ok, that does exactly what I needed. I thought it was not already implemented. Thanks!

[HamedHemati]

We can set this discussion as answered.

[prateeky2806]

Hi @AntonioCarta, when we use the function to split the train stream into train and valid stream, Then which transformation is the validation stream using by default? How do we change the used transformation for a whole stream to say 'eval' transform. I tried to set the transformation for individual datasets in the valid stream but it seems like the tranform_groups is set to None for them. A complete thread for this issue is [here].(#995 (comment))
benchmark = benchmark_with_validation_stream(benchmark_original, validation_size=0.2, input_stream='train')