PASCAL VOC transforms doen't work proprietly

[adrianosantospb]

Describe the bug

Hi,

I'm trying to implement the transform compose to SSD model using PASCAL VOC dataset. I'm doing:

IMAGENET_MEAN = 0.485, 0.456, 0.406  # RGB mean
IMAGENET_STD = 0.229, 0.224, 0.225  # RGB standard deviation
IMAGE_SIZE = 300

transform_train = A.Compose([
    A.Resize(IMAGE_SIZE, IMAGE_SIZE),
    A.RandomCrop(width=IMAGE_SIZE, height=IMAGE_SIZE),
    A.ColorJitter(brightness=0.6, contrast=0.6, saturation=0.6, hue=0.6, p=0.4),
    A.HorizontalFlip(p=0.5),
    A.Blur(p=0.1),
    A.CLAHE(p=0.1),
    A.Posterize(p=0.1),
    A.ToGray(p=0.1),
    A.ChannelShuffle(p=0.05),
    A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
    ToTensorV2(),
], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['labels']))

My dataset class is:

import torch
from torch.utils.data import Dataset
from PIL import Image
import numpy as np
import json

class VOCDataset(Dataset):
    def __init__(self, images_files_path, objects_files_path, transform):
        
        self.transform = transform

        #read data file from json file
        with open(images_files_path, 'r') as i:
            self.images = json.load(i)
        
        with open(objects_files_path, 'r') as o:
            self.objects = json.load(o)
            
        assert len(self.images) == len(self.objects)
        
    def __len__(self):
        return len(self.images)
    
    def __getitem__(self, idx):
        
        image = Image.open(self.images[idx], mode= "r")
        image = image.convert("RGB")
        image = np.array(image)

        #Read objects in this image
        objects = self.objects[idx]

        bboxes = objects["boxes"]
        labels = objects['labels']
        difficulties = torch.ByteTensor(objects['difficulties'])

        #Apply transforms
        transformed = self.transform(image=image, bboxes=bboxes, labels=labels)
        
        image = transformed['image']
        bboxes = transformed['bboxes']
        labels = transformed['labels']

        image = torch.FloatTensor(image)
        bboxes = torch.FloatTensor(bboxes)
        labels = torch.LongTensor(labels)

        return image, bboxes, labels, difficulties

    def collate_fn(self, batch):

            images = list()
            boxes = list()
            labels = list()
            difficulties = list()

            for b in batch:
                images.append(b[0])
                boxes.append(b[1])
                labels.append(b[2])
                difficulties.append(b[3])

            images = torch.stack(images, dim=0)

            return images, boxes, labels, difficulties

Durring the model trainig, I got a ruge loss value (something likes 10000). But when I use a simple transform (just crop and normalize with python code), the model learn very well.

My train part is:

momentum = 0.9
lr = 1e-3
weight_decay = 5e-4
num_epochs = 10

optimizer = optim.SGD(model.parameters(), lr=lr, weight_decay=weight_decay, momentum = momentum)
scheduler = OneCycleLR(optimizer, max_lr= lr, epochs = num_epochs, steps_per_epoch = 2*(len(train_dataloader)), 
                        pct_start=0.3, div_factor=10, anneal_strategy='cos')

criterion = MultiBoxLoss(model.default_boxes).to(device)

def train(train_loader, model, criterion, optimizer, epoch):
    """
    One epoch's training.

    :param train_loader: DataLoader for training data
    :param model: model
    :param criterion: MultiBox loss
    :param optimizer: optimizer
    :param epoch: epoch number
    """
    torch.cuda.empty_cache()
    model.train()  # training mode enables dropout

    losses = AverageMeter()  # loss

    # Batches
    for _, (images, boxes, labels, _) in tqdm(enumerate(train_loader), total=len(train_loader)):
        images = images.to(device)  # (batch_size (N), 3, 300, 300)
        boxes = [b.to(device) for b in boxes]
        labels = [l.to(device) for l in labels]
        
        # Forward prop.
        predicted_locs, predicted_scores = model(images)  # (N, 8732, 4), (N, 8732, n_classes)

        # Loss
        loss = criterion(predicted_locs, predicted_scores, boxes, labels)  # scalar

        optimizer.zero_grad()

        # Backward prop.
        loss.backward()

        # Update model
        clip_gradient(optimizer, 1)
        optimizer.step()

        losses.update(loss.item(), images.size(0))

        # adjust learning rate
        if scheduler is not None:
            scheduler.step()

    # Print status
    print('Loss {loss.val:.4f} ({loss.avg:.4f})\t'.format(loss=losses))
    del predicted_locs, predicted_scores, images, boxes, labels  # free some memory since their histories may be stored



torch.cuda.empty_cache()

for epoch in range(num_epochs):

    print("Epoch {}".format(1 + epoch))

    train(train_loader=train_dataloader,
            model=model,
            criterion=criterion,
            optimizer=optimizer,
            epoch=epoch)

[ternaus]

It is hard to debug if there is no way to reproduce the issue.

Could you please visually check that after the augmentations, the pipeline bounding boxes are where they should be?

If this is not the case and you can provide an example of images + bounding boxes that were processed incorrectly, we will be happy to fix the issue ASAP.

But if bounding boxes are where they should be, the issue with convergence will be in the later phase of training.

[ternaus]

@adrianosantospb feel free to reopen when you will have a minium example that reposdices the issue