[QUESTION] How to map semantic labels of another dataset and predictions coming from a segmentation model in Hydra?

[puren]

Hello,

We want to use Hydra with ade20k dataset with its ground truth semantic labels and also prediction coming from another segmentation model, i.e., paddleseg mobile trained over ade20k 150 classes dataset. Hence we want to understand how to use that dataset in Hydra and how to map the segmentation ground truth and prediction labels based on ade20k ground truth labels to Hydra label space. We are following uhumans2 dataset example for this. There we see that Hydra\config\label_spaces<label_space_name>_label_space.yaml, Hydra\config\label_remaps<label_space_name>.yaml and Hydra-ros\hydra_ros\config\color<label_space_name>.csv files are used. We see files relevant to ade20k dataset in these folders, but we have some questions how to use them exactly.

1. First question is about sub/super ids. For instance, book/bookshelf related objects are labeled as super id 2 in uhumans2 and there are multiple sub ids for the same types of objects apparently inside uhumans2, such as 5, 15 for book/bookshelfs like objects based on label_spaces data. As we understand it is a remapping of dataset specific labels to more generic labels inside Hydra. Is our understanding correct?
2. If so, is there a general list of super ids? Else are the super ids we see in Hydra\config\label_remaps\uhumans2_office.yaml specific for uhumans2? Can we use the same super id mapping for ade20k? To do that which files to edit exactly? All Hydra\config\label_spaces<label_space_name>_label_space.yaml, Hydra\config\label_remaps<label_space_name>.yaml and Hydra-ros\hydra_ros\config\color<label_space_name>.csv files?
3. An example of remapping based on understanding is as follows: In Hydra/config/label_spaces/ade20k_full_label_space.yaml, book is labeled as 67, while in uhumans2 books/bookshelf related objects are labeled as super id 2. So if we leave ade20k labeling as it is, would it would understand it at all in hydra while constructing the environment? Else we should treat 2 as super id for books, then books in ade20k full dataset should be mapped to 2 in remapping and .csv files? Is our understanding correct?
4. In uhumans2.launch, semantic_label_remap_filepath that is Hydra\config\label_spaces<label_space_name>_label_space.yaml is called based on a condition of use_single_channel_semantic_map and it is set to false by default. So is this set to false, since uhumans2 dataset publishes semantic segmentation as rgb images by default? Then for another dataset or segmentation prediction that is in semantic map format as single channel, we should set this to True and create a <label_space_name>_label_space.yaml file accordingly?
5. We want to use the whole 150 labels ade20k but we see separate label space and color .csv file for that. Then we should create label_space, label remapping and .csv file for that right?
6. Also, one final question about segmentation topic: To get segmentation, according to uhumans2 dataset, we see that label_topic is subscribed to /semantic/image_raw and /tesse/seg_cam/converted/image_raw (for single channel label like semantic map?). So if we want to get the predictions or ground truth from another dataset, we should publish them to the same topics, right?

I hope our questions are clear. If we misunderstand anything, please let us know.

Best regards,
Püren