A batch without any targets causes the script to fail

[raimannma]

🐛 Bug

To Reproduce (REQUIRED)

Steps to reproduce the behavior:

Default training, but use a dataset that has images without any bounding boxes.

Expected behavior

Do not throw any error.

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.8.0-59-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 465.19.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.0
[pip3] pytorch-lightning==1.3.0rc1
[pip3] torch==1.9.0+cu111
[pip3] torch2trt-unofficial==0.0.3
[pip3] torchaudio==0.9.0
[pip3] torchfile==0.1.0
[pip3] torchmetrics==0.4.0
[pip3] torchvision==0.10.0+cu111
[pip3] torchviz==0.0.2

Additional context

The error is raised in line 106 of models/transform.py.
Because torch.cat can't process an empty list.

[zhiqwang]

Hi @raimannma , I found that you file a PR #129 for this problem, we are welcome for fixing / enhancing this features about training with negative samples.

At a quick glance, a batch containing no targets will also affect the calculation in assign_targets_to_anchors, we should also adjust the calculations here.

https://github.com/zhiqwang/yolov5-rt-stack/blob/82d6afbfc6286ec4e01c9a75730b4c9b9c90fd86/yolort/models/box_head.py#L177

BTW, I found there are some bugs in the part of loss computation, and I'm trying to fix it in #125

[raimannma]

Yeah, I opened a PR.

But some more tests showed that this PR isn't fully fixing the bug.