After mmgrounding-dino-t model trained my own dataset, the test labels became chaotic.

[h030162]

I have defined a great many types of human behaviors, including sleeping, raising hands, standing, eating, smoking and so on, more than 20 in all. After I used the mmgrounding-dino-t model to finetune my own dataset, I found that even on the data that had been trained, the inferred labels were in a mess. Moreover, I also found that there were two or more different labels under the same coordinates. What's the cause of this?
The following is a schematic diagram. On the left is the ground truth of the annotation, and on the right is the detection value after the model has been trained.

output is：
{'bbox': [742.95, 368.44, 904.14, 584.8], 'score': 0.55, 'label': 18, 'category': 'raising right'}
{'bbox': [742.95, 368.44, 904.14, 584.8], 'score': 0.48, 'label': 17, 'category': 'raising left'}
However, I only annotated one behavior of sleeping on this picture, and this picture belongs to the training set.
Why does this phenomenon occur?