Help Wanted 🙏 Add Oriented Bounding Boxes (OBB) Task with DOTA v2 Training

[glenn-jocher]

Hello YOLOv8 Enthusiasts,

We are reaching out to the talented community to help add support for DOTA v2 training with Oriented Bounding Boxes (OBB) for our YOLOv8 repository. This addition will notably enhance our detection capabilities for remote sensing imagery, enabling superior performance for real-world applications.

Related to issue #624 and PR #1977

Project Description:

Our primary objective with this issue is to integrate the DOTA v2 dataset into our YOLOv8 training pipeline, with a focus on Oriented Bounding Boxes. This demands crafting methods or functions to accommodate the unique data format of DOTA v2, and seamlessly incorporate it into our existing training framework.

Example code:

from yolov8 import YOLO

# Initialize a model
model = YOLO('yolov8n-obb.pt')  # load a pretrained OBB model

# Train the model on DOTA v2 with OBB
model.train(data='DOTAv2.yaml', epochs=30, imgsz=640)

In this envisaged example, the data argument would accept a configuration file specifically crafted for the DOTA v2 dataset with OBB, 'DOTAv2.yaml'. This configuration file should encompass all essential details to fetch, process, and utilize the dataset during training.

Expected Deliverables:

• Code that processes and manages the DOTA v2 dataset with Oriented Bounding Boxes.
• Comprehensive test cases to validate the flawless functioning of the added feature.
• Amended documentation, enlightening users on the ins and outs of this fresh capability.

How to contribute:

1. Express your interest in this endeavor by commenting on this issue.
2. Fork the YOLOv8 repository.
3. Generate a fresh branch in your forked repository.
4. Execute the necessary code modifications.
5. Develop tests that corroborate the effectiveness of your code.
6. Revise the README and affiliated documentation accordingly.
7. Submit a detailed pull request, illuminating your changes, their relevance, and how they enhance the repository.

Every contribution, regardless of its magnitude, is deeply appreciated. Your efforts contribute massively to the broader community, and every input is treated with respect.

Community Support:

To amplify its visibility, this issue will be pinned. If you need support or have inquiries, do not hesitate to contact us either on this thread or via our Discord.

Gratitude and best wishes on your coding journey!

Warm Regards,
The YOLOv8 Team

[atanasko]

Hi @glenn-jocher,

I already face the OBB problem with WOD (Waymo Open Dataset) :-). My code with WOD I think already works

but with OBB I have

so I can not train the model corectly.

Can you give me a hint what part of the code I should have a look in order to add orientation to the bounding box?

Thanks in advance,
Atanasko

[ZuyongWu]

extremely waiting for it........

[glenn-jocher]

@atanasko well the good news is that we have an open OBB PR in #1977 and I've separately added support for DOTAv2 dataset training in #4258.

The main problem is the OBB PR #1977 was open for a while and now the original author isn't responding. If you could help us get it going that would be super helpful.

Also if you could help us get an OBB version of the WOD dataset that would be awesome too!

[atanasko]

hi @glenn-jocher,

I'll have a look and come back to you.

Best regards,
Atanasko

[glenn-jocher]

Hi @atanasko,

That sounds excellent! We truly appreciate your willingness to delve into this. Your insights and contributions could potentially be pivotal to advancing this feature. Please feel free to share any questions, concerns, or progress updates - we are here to provide any guidance you may need.

Looking forward to your findings!

Best Regards,
Glenn Jocher & The Ultralytics Team

[atanasko]

hi @glenn-jocher,

I grab obb-dev branch into my fork of the ultralitics repository. I will download DOTA v2 dataset and try to start training process.

Best regards,
Atanasko

[glenn-jocher]

@atanasko got it! There's probably some changes that may need to be made to label formats, as DOTAv2 dataset is in the new YOLO OBB format I defined in https://docs.ultralytics.com/datasets/obb/, each row is class_index, then the 4 normalized points xy1, xy2, xy3, xy4

[atanasko]

hi @glenn-jocher,

In the ticket description example code

# Initialize a model
model = YOLO('yolov8n-obb.yaml')  # load an OBB model

initialization should be from model .pt file no?
ex.

# Initialize a model
model = YOLO('yolov8n-obb.pt')  # load an OBB model

[glenn-jocher]

@atanasko yes this is True, I've updated it! Note we don't have any pretrained models yet, this is the intended use case once we have them.

[atanasko]

hi @glenn-jocher,

I download the dataset and prepare locally as advised in DOTA-v2.0 instructions, and try to trigger the process, but there were warnings in the log file

and when I open ex. P0000.txt label txt file labels are like:

2238.0 1791.0 2254.0 1791.0 2254.0 1813.0 2238.0 1813.0 small-vehicle 1
1462.0 2131.0 1472.0 2131.0 1472.0 2145.0 1462.0 2145.0 small-vehicle 1
1122.0 1607.0 1142.0 1607.0 1142.0 1620.0 1122.0 1620.0 small-vehicle 0
146.0 1860.0 164.0 1860.0 164.0 1871.0 146.0 1871.0 small-vehicle 1
211.0 1815.0 231.0 1815.0 231.0 1823.0 211.0 1823.0 small-vehicle 1
170.0 1826.0 189.0 1826.0 189.0 1836.0 170.0 1836.0 small-vehicle 1
80.0 2245.0 99.0 2245.0 99.0 2263.0 80.0 2263.0 small-vehicle 1
220.0 2470.0 233.0 2470.0 233.0 2490.0 220.0 2490.0 small-vehicle 1
172.0 2491.0 186.0 2491.0 186.0 2513.0 172.0 2513.0 small-vehicle 1
1158.0 1137.0 1183.0 1137.0 1183.0 1152.0 1158.0 1152.0 small-vehicle 1

and this is not like format expected in OBB.

Any advice how to proceed with this?

Thanks in advance,
Atanasko

[atanasko]

hi @glenn-jocher,

OK, by using the following scrip I process all label files.

from pathlib import Path
import sys

files = Path(sys.argv[1] ).glob('*')
for file_name in files:
    file = open(file_name, "r")
    replaced_content = ""

    for line in file:
        line = line.strip()
        replaced_content = replaced_content + line[-1] + " " + line[:line.find(next(filter(str.isalpha, line)))] + "\n"
    file.close()

    write_file = open(file_name, "w")
    write_file.write(replaced_content)
    write_file.close()

and try to trigger the training process, but I have the following error in the log

Any advice?

[glenn-jocher]

@atanasko don't worry, you don't need to do any manual updates to the dataset, just run this command and the dataset will automatically download and convert to the YOLO OBB format:

yolo train model=yolov8n.pt data=DOTAv2.yaml

Note training will crash since OBB is not a supported task yet, but the dataset itself will download for you to look at.

[glenn-jocher]

Yo super surprise guys, the YOLO OBB dataset format I settled on by complete coincidence, and much to my amusement, is trainable by Segmentation models also!!!!

Here's a DOTAv2 mosaic training in Segment mode:

[atanasko]

hi @glenn-jocher,

I do try using the downloading script it downloads only DOTAv1 images no? Should v2 images be downloaded also?

Thanks in advance,
Atanasko

[glenn-jocher]

@atanasko the training script with download DOTAv2 images:

yolo train model=yolov8n-seg.pt data=DOTAv2.yaml

[atanasko]

hi @glenn-jocher,

I grab your changes and try to start the training process by

from ultralytics import YOLO

# Initialize a model
model = YOLO('yolov8n-obb.yaml')  # load an OBB model
# model = YOLO('yolov8n.pt')  # load an OBB model

# Train the model on DOTA v2 with OBB
model.train(data='DOTAv2.yaml', epochs=30, imgsz=640, batch=32)

for a new model, but in the

~/ultralitics/data/utils.py

there is

def img2label_paths(img_paths, use_obb):
    """Define label paths as a function of image paths."""
    # /images/, /labels/ substrings
    sa, sb = f'{os.sep}images{os.sep}', f'{os.sep}labels{os.sep}' if not use_obb else f'{os.sep}labelTxt{os.sep}'
    return [sb.join(x.rsplit(sa, 1)).rsplit('.', 1)[0] + '.txt' for x in img_paths]

labels are expected in "~/labelTxt" sub-directory. Is this correct?

[atanasko]

hi @glenn-jocher,

And also in

~/ultralitics/data/utils.py

def verify_image_label(args):
...
                elif obb:
                    assert lb.shape[1] == 6, f'labels require 10 columns, {lb.shape[1]} columns detected'
                    assert (lb[:, 1:] <= 1).all(), \
                        f'non-normalized or out of bounds coordinates {lb[:, 1:][lb[:, 1:] > 1]}'
                    assert (lb >= 0).all(), f'negative label values {lb[lb < 0]}'
...

fails

[glenn-jocher]

@atanasko PR in non-operative currently, your example shows intended usage once it's complete.

[atanasko]

hi @glenn-jocher,

thanks! how can I be of any further help here? I see that there is already new obb branch in the ultralytics repository

[glenn-jocher]

Hi @atanasko,

Thanks for offering your help! We definitely appreciate community's contribution. At this stage, we're still working on the initial implementation of the OBB feature. As we progress, we'll mark this issue with appropriate tags like 'help wanted' or 'good first issue' whenever we identify areas that can benefit from external contributions.

These tasks might include creating unit tests to validate the feature or enhancing the documentation to explain the new functionality. When those tasks are ready, we'll certainly welcome your assistance. In the meantime, feel free to browse other issues to see if there's something else where you would like to contribute.

Thank you again for your interest in this project, and we're looking forward to collaborating with you in the future!

Best,
YOLOv8 team

[ghost]

It seems that some variables are not defined in loos.py.

[glenn-jocher]

@xiaofulee thank you for reporting this issue.

From what I can see, it seems like there might be some misunderstandings with variable assignments in loss.py. The error message you're seeing typically arises when a variable is referenced before it has been assigned any value.

It would be beneficial to ensure that the all variables are defined and assigned before they are called. This could be a version error, or possibly some alterations in the code that lead to this problem.

We'll look into this issue promptly and try to get it sorted out as soon as possible.

Thank you for your patience and support.

Best,
The YOLOv8 Team

[glenn-jocher]

@TuanTorushiro At present, neither YOLOv5 nor YOLOv8 officially support oriented bounding boxes (OBB). We're actively exploring the inclusion of OBB in future updates. Stay tuned with the repo for any announcements on this feature! 🌟

[dev-taewan]

"I am using a modified version with a 360-degree angle from https://github.com/thohemp/cube_detector. Although it's a variant of v5.0, it works well." 2023년 12월 11일 (월) 오전 9:38, Luu Duc Tuan ***@***.***>님이 작성:

…

@TuanTorushiro <https://github.com/TuanTorushiro> hi there! Currently, YOLOv8 predicts axis-aligned bounding boxes by default, which do not include orientation information. Oriented bounding boxes (OBB) are a feature on our roadmap that would allow us to estimate the orientation of objects, but it is not yet implemented in the official Ultralytics YOLOv8 repo. For now, if you need to work with oriented bounding boxes, you would need to develop custom functions that can predict the rotation angle and adjust your training pipeline accordingly. Keep an eye on the repo for future updates on OBB support. 🔄 Happy coding! "So, does YOLOv5 support oriented bounding boxes (OBB) in its version?" — Reply to this email directly, view it on GitHub <#4242 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHWRTZLKFKZCT3RFERH4SE3YIZIZXAVCNFSM6AAAAAA3JHDKIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGE2TEOJYG4> . You are receiving this because you commented.Message ID: ***@***.***>

[TuanTorushiro]

"I am using a modified version with a 360-degree angle from https://github.com/thohemp/cube_detector. Although it's a variant of v5.0, it works well." 2023년 12월 11일 (월) 오전 9:38, Luu Duc Tuan @.>님이 작성:
…
@TuanTorushiro https://github.com/TuanTorushiro hi there! Currently, YOLOv8 predicts axis-aligned bounding boxes by default, which do not include orientation information. Oriented bounding boxes (OBB) are a feature on our roadmap that would allow us to estimate the orientation of objects, but it is not yet implemented in the official Ultralytics YOLOv8 repo. For now, if you need to work with oriented bounding boxes, you would need to develop custom functions that can predict the rotation angle and adjust your training pipeline accordingly. Keep an eye on the repo for future updates on OBB support. 🔄 Happy coding! "So, does YOLOv5 support oriented bounding boxes (OBB) in its version?" — Reply to this email directly, view it on GitHub <#4242 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWRTZLKFKZCT3RFERH4SE3YIZIZXAVCNFSM6AAAAAA3JHDKIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGE2TEOJYG4 . You are receiving this because you commented.Message ID: @.>

"Could you show me how the directory structure for the training, validation, and test datasets is organized?" I am training in GPU Cuda window11

[glenn-jocher]

@TuanTorushiro, it's great to hear that you've found a working solution for OBB with a YOLOv5 variant. As for your dataset organization, typically, you want to structure your directories with separate folders for your training, validation, and test sets. Each set should have a directory of images and a corresponding annotation file or directory of annotation files, depending on the format required by the training script. Keep all images and annotations organized, ensuring paths are correctly referenced in any configuration or data files used during training. 📁🚀

[dev-taewan]

train - images - labels val -images - labels test - images - labels It is identical to YOLO, except that an 'angle' has been added to the label at the end. Label content: 0 0.2462 0.7654 0.0442 0.2375 337

…

Message ID: ***@***.***>

[Zzzxi]

火车 -图像 -标签 瓦尔 -图像 -标签 测试 -图像 -标签 它与YOLO相同，只是在标签上添加了一个“角度” 在最后。标签内容：0 0.2462 0.7654 0.0442 0.2375 337
...
邮件 ID：@.***>

Your tag is not formatted correctly, you should refer to this.https://docs.ultralytics.com/datasets/obb/

[glenn-jocher]

@Zzzxi, your directory structure looks correct for training a YOLO model, with separate folders for images and labels. The addition of an angle to the label signifies your move towards using oriented bounding boxes (OBB). Make sure that your custom labels follow the expected format and that any custom code modifications can correctly interpret the angle for training. Good luck with your project! 📐👍

[W-Hoole]

when training using the OBB branch, i noticed that my MAP is about 90% but my oriented bounding boxes are not oriented well. Could this have to do with the conversion from xyxyxyxy to xywhr ? i have formatted my labels to cls x1 y1 x2 y2 x3 y3 x4 y4 starting at the top left point and going clockwise. Are there any solutions why my model is not learning correctly but my MAP is so hight?

[Laughing-q]

@W-Hoole the obb branch is still working on progress and be updated every day, probably some commits breaks your training or inference.
We'll officially launch obb task soon! :)

[Zzzxi]

I encountered the following issues while using OBB:

1. The progress bar remains at 0%.
2. The final result is not satisfactory. (The dataset I used is self-made and there are no issues with it.)

What could be the reasons for these issues? If the project is still in progress, I hope the problems I encountered can help you further improve the OBB project.

[Zzzxi]

Interestingly, despite the less-than-ideal results obtained through val.py, the predicted images generated by detect.py are outstanding! What could be the reason for this?

[Laughing-q]

@Zzzxi wow! that's good to know!
yes current calculation of metrics might result an inconspicuous metric, cause we don't use the accurate way to calculate iou for rotated objects as that might get cuda or c++ involved(which might make the installation become difficult for users) if we want a fast version of that. Currently we're just using a suboptimal replacement of that.

[Zzzxi]

@Zzzxi wow! that's good to know! yes current calculation of metrics might result an inconspicuous metric, cause we don't use the accurate way to calculate iou for rotated objects as that might get cuda or c++ involved(which might make the installation become difficult for users) if we want a fast version of that. Currently we're just using a suboptimal replacement of that.

I look forward to your project becoming more refined!

[Yuri-Njathi]

Very Interesting work! I'd love to help where I can with adding oriented bounding boxes task with DOTA v2 Training

[glenn-jocher]

@Yuri-Njathi that's wonderful to hear! We appreciate your interest in contributing to the OBB task with DOTA v2 training. I suggest starting by getting familiar with our guidelines on contributing, then setting up a development environment. After that, you can fork the YOLOv8 repo, create a new branch for your work, and experiment with integrating the OBB functionality. Remember to check the existing issues and PRs to avoid duplicating work. Your involvement is highly valued, and we look forward to your innovative solutions. Happy coding! 🚀🧠

[TuanTorushiro]

@TuanTorushiro, it's great to hear that you've found a working solution for OBB with a YOLOv5 variant. As for your dataset organization, typically, you want to structure your directories with separate folders for your training, validation, and test sets. Each set should have a directory of images and a corresponding annotation file or directory of annotation files, depending on the format required by the training script. Keep all images and annotations organized, ensuring paths are correctly referenced in any configuration or data files used during training. 📁🚀

Although I haven't been able to draw Oriented Bounding Boxes (OBBs) around objects yet, I have found a solution to determine the orientation for each default bounding box. Determining the orientation is crucial as it precisely defines the rotation angle of the object. This is a fundamental prerequisite for developing applications in the field of automated robotic object grasping and lifting. I hope Yolov8 will soon release a version that supports Oriented Bounding Boxes!

[glenn-jocher]

@TuanTorushiro, your anticipation for OBB support in YOLOv8 and its potential applications in robotics is encouraging! The team is diligently working on this feature, and your insights on orientation will be very useful for this development. Keep an eye on our updates for the upcoming release. Your patience and input are much appreciated! 🔄✨

[Zzzxi]

@Zzzxi wow! that's good to know! yes current calculation of metrics might result an inconspicuous metric, cause we don't use the accurate way to calculate iou for rotated objects as that might get cuda or c++ involved(which might make the installation become difficult for users) if we want a fast version of that. Currently we're just using a suboptimal replacement of that.

The performance metrics obtained from training my own dataset are still not ideal. Is the IOU still being further improved? @Laughing-q

[Zzzxi]

In the latest version of the OBB dataset, I found that using train.py does not run properly, as shown in the image.

[glenn-jocher]

@Zzzxi we're aware and actively working on improving OBB support, including refining the IOU calculations. Your patience and feedback are essential to our progress. Stay tuned for updates! 🔄🛠️ #OBBUpdate

[prashant-dn]

Following up in this direction,

@glenn-jocher

I saw that you have used the segmentation head to implement the obb training, and I see that the definition of the v8OBBLoss in the obb branch is based on the v8SegmentationLoss.

Are you planning to use the segment method to implement the obb task? I personally don't think that's a good idea. obb is a problem between hbb (horizontal bounding box) and Segmentation. It is true that the rotated rectangle can be regarded as a special quadrilateral segmentation mask, but if you do that, this obb branch is unnecessary. We may only need to add limit on the number of mask edges or vertices in Segmentation somehow.

In addition, in the head of OBB, you convert 0-1 theta to -180 to +180 degrees. So you're going to implement 360 degree precise obb detection, not 180 degree. To put it simply, for example, for a car in an aerial image, the obb branch is intended to distinguish the direction of its front and rear? While it seems this part of annotation do not exist in DOTA.

So would you describe your development plan and algorithm ideas, which will help the community to contribute together. Thank you!

As @surefyyq mentioned, using segmentation for obb seems unfit here. It would be better to do this for an algo that solely focuses on segmentation and is probably SOTA.

Can someone (@Laughing-q) add his thoughts on this?

[glenn-jocher]

@prashant-dn the ongoing development of OBB is exploring various approaches, and while the segmentation head concepts provided insights, it's clear that we need a more tailored solution for OBB. The angles and unique challenges presented by OBB require specific handling beyond typical HBB or segmentation. Optimization and annotation clarity are also critical considerations. Rest assured, we're actively working on refining the process and welcome further community input and discussion on the best path forward. 🔄🎯

[TuanTorushiro]

@glenn-jocher,
"I see YOLO is focusing on image processing. What do you think about extending its development into audio processing? Or could you suggest some frameworks for handling this well?"

[glenn-jocher]

@TuanTorushiro yOLO specializes in computer vision tasks, and we currently don't have plans to extend into audio processing. For audio-related tasks, you might want to explore frameworks like TensorFlow or PyTorch, which have audio processing capabilities. 🎵👨‍💻

[SwEngine]

@glenn-jocher What do you think about supporting gray scale images with one channel? There are many people wanting this feature. 3 channels are hard to process in CPUs.

[TuanTorushiro]

@TuanTorushiro yOLO specializes in computer vision tasks, and we currently don't have plans to extend into audio processing. For audio-related tasks, you might want to explore frameworks like TensorFlow or PyTorch, which have audio processing capabilities. 🎵👨‍💻

Dear @glenn-jocher,
I am a programmer, I'm using the YOLOv8 model to train datasets on CUDA and export ONNX files. I've read about the GNU Affero General Public License v3.0. So, when I develop an application using Python specifically for internal use within my company and use the YOLOv8 model, am I allowed to do so? Would this violate Ultralytics' copyright policies? Could you please explain this further to me?