CUDA error: device-side assert triggered

[JingyuLi-code]

When I default parameter: python train.py --gpus 0 --id experiment-xe --geometry_relation True , it can run but in 6 epoch around errors occur as follows:
I guess that 1. I use wrong dataset label. 2. code error. How can I fix this issue? Thanks!

/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1367,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1367,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1367,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [1367,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "train.py", line 155, in
train(opt)
File "train.py", line 87, in train
out = decoder(sg_data, fc_feats, att_feats, labels, att_masks)
File "/home1/ljy/.conda/envs/relation_trans/lib/python2.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home1/ljy/VSUA-Captioning-master/models/CaptionModel.py", line 25, in forward
return getattr(self, '_'+mode)(*args, **kwargs)
File "/home1/ljy/VSUA-Captioning-master/models/VSUAModel.py", line 284, in _forward
if sample_mask.sum() == 0:
RuntimeError: CUDA error: device-side assert triggered