cope with none hoi image?

[dragen1860]

Dear author:
recently i did some modification on hico dataset, remove some of the original 117 hoi categories, say 20 categories only. Therefore there are no hoi annotation on some images.
When i try to train the modified hico dataset, I get an error:

File "/media/sdf/long/hoi/qpic/models/hoi.py", line 226, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/media/sdf/long/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/sdf/long/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/media/sdf/long/hoi/qpic/models/matcher.py", line 119, in forward
    cost_verb_class = -(out_verb_prob.matmul(tgt_verb_labels_permute) / \
RuntimeError: mat1 dim 1 must match mat2 dim 0

I debugged into code and realise that for no hoi annotation image, the tensor in target will be emtpy tensor, such as

I guess this should be the reason of above error.

I wonder does the code support none-hoi-annotation images? if not , how to make it supported ?