OnlyPlanes: Incrementally Tuning Synthetic Training Datasets for Satellite Object Detection

Project Links: [Paper] [Project Page] [Video] [Blog]

Abstract: This paper addresses the challenge of solely using synthetic data to train and improve computer vision models for detecting airplanes in satellite imagery by iteratively tuning the training dataset. The domain gap for computer vision algorithms remains a continued struggle to generalize when learning only from synthetic images and produce results comparable to models learning from real images. We present OnlyPlanes, a synthetic satellite image training dataset of airplanes in a top-view that contains 12,500 images, 132,967 aircraft instances, with 80 fine grain attributes. We detail the synthetic rendering process to procedurally-generate and render training images and introduce the synthetic learning feedback loop that iteratively finetunes the training dataset to improve model performance. Experiments show the prior performance degradations and false positives when testing the model on real-world images were corrected when incrementally incorporating new features to the training dataset such as diverse plane formations and altitude variations. We demonstrate the promising future for continually improving models by tuning the synthetic training data developed in an iterative fashion to effectively close the synthetic to real domain gap. The OnlyPlanes dataset, source code and trained models are available at https://github.com/naivelogic/OnlyPlanes.

Example Inference Results from OnlyPlanes Models tested on Real-World Datasets

Dataset Overview

OnlyPlanes is a small dataset (less than 50k images and less than 20GB in weight) for airplane object and instant segmentation computer vision task from satellite imagery.

Dataset	Size	Description
OnlyPlanes.zip	14.8 GB	Training
binary_plane_annotations.json	1.57 GB	Labels
civil_role_annotations.json	506.43 MB	Labels
Bing2D_Empty.zip	531.8 MB	Training
Bing2Dempty_annotations.json	0.137 MB	Labels

Dataset Description

The OnlyPlanes dataset contains 12,500 images and 132,967 instance objects consisting of four categories (plane, jumbo jet, military, helicopter) with 80 fine-grain attributes that define the plane model (e.g., Boeing 737). A single training dataset is provided for both object detection and instance segmentation tasks at 1024x1024 image resolution using ten different airport.

Models Overview

The training methodology implemented involved a Faster-RCNN and a Mask-RCNN model solely trained on the OnlyPlanes synthetic dataset. The Faster-RCNN model is used for object detection and the Mask-RCNN model is used for instance segmentation.

Name	Model Zoo Config	Trained Model	Train Config
OnlyPlanes Object Detection	faster_rcnn_R_50_FPN_3x	onlyplanes_faster_rcnn_r50-0034999.pth	config.yaml
OnlyPlanes Instance Segmentation	mask_rcnn_R_50_FPN_3x	onlyplanes_mask_rcnn_r50-0054999.pth	config.yaml

Benchmarking

To evaluate the performance of OnlyPlanes we tested the performance on detecting airplanes in other notable datasets. Benchmark datasets include: RarePlanes Real Test, DIOR Test, iSAID Val, and NWPU VHR-10.

Benchmark Dataset	mAP	mAP50	AR	OnlyPlanes Model
RarePlanes	59.10	91.10	65.40	Faster R-CNN
NWPU VHR10	73.66	98.32	78.90	Faster R-CNN
NWPU VHR10	42.53	98.25	47.61	Mask R-CNN
DIOR	48.56	82.73	57.48	Faster R-CNN
iSAID	46.48	68.99	53.89	Faster R-CNN
iSAID	20.56	57.74	30.25	Mask R-CNN

Download and Prepare Benchmark Datasets

To test the performance of the model the below benchmark datasets were used.

• iSAID | paper | dataset | binary plane json

  • Note: We used the pre-processed version of the dataset from CATNet approach where images were split into 512 x 512 patches and extreme aspect ratios from the official toolkit were corrected.
  • Test dataset statistics: 11,752 images | 6,613 airplane instances from the validation dataset.

• RarePlanes | paper | dataset | binary plane json

  • Note: for evaluation only RarePlanes Real Test dataset was used. Instructions to download the RarePlanes dataset RarePlanes Public User Guide. Additionally, since the official RarePlanes repository is no longer available (summer 2022), refer to the unofficial mirror repo.
  • Test dataset statistics: 2,710 images and 6,812 airplane instances from the real test dataset.

• NWPU VHR10 | paper | dataset | binary plane json

  • Test dataset statistics: 650 images and 757 airplane instances from the positive image set.

• DIOR | paper | dataset | binary plane json

  • Note: we first converted the DIOR VOC to COCO annotations using only the we used the test.txt Horizontal Bounding Boxes annotations.
  • Test dataset statistics: 2,932 images and 8,042 airplane instances from the test dataset.

Here is the file structure used for these benchmark datasets

Benchmark Datasets
├── data
│   ├── dior
│   │   ├── Annotations
│   │   ├── ImageSets
│   │   ├── JPEGImages-test
│   │   └── JPEGImages-trainval
│   ├── rareplanes
│   │   └── real
|   │       ├── metadata_annotations
|   |       │   ├── instances_test_aircraft.json
|   |       │   └── instances_test_role.json
|   │       └── test
|   |           └── PS-RGB_tiled
│   ├── isaid
│   │   ├── annotations
│   │   |   ├── instances_val.json  # original - not used
|   │   │   └── val_binary_plane_coco_annotations_ALL.json # used for benchmarking
│   │   └── val
    └── nwpu_vhr10
        ├── ground truth
        └── positive image set

---

Getting Started

This step we install all the necessary dependencies to train an object detector on a local machine. For this workshop we will be utilizing detectron2 that runs on PyTorch as the framework for training DNNs. Quick start

• Step 1: clone workshop repo (git required): git clone https://github.com/naivelogic/OnlyPlanes.git
• Step 2: navigate to repo folder: cd OnlyPlanes
• Step 3: install dependencies with Anaconda: sh scripts/00_install_env.sh
• Step 4: verify installation: conda activate OP

Usage: Local vs AML

For this repository to simplify the usage, we will provide the code from a local computational perspective. In the paper, we utilized Azure Machine Learning for training and evaluating the performance of the models. Additionally, in the paper all data was stored on an Azure blob container. While Azure ML is great for scaling compute intensive workloads, as long as you meet the requirements below a single GPU can put utilized to reperform results.

Requirements

• Python > 3.5
• Pytorch > 1.9
• torchvision that matches the PyTorch installation. You can install them together at pytorch.org to make sure of this.
• Cuda > 11.0
• Detectron2
• Make sure the CUDA, Pytorch and Detectron2 have the correct CUDA version if using GPU

special notes on getting started with requirements

My OS for this project was a ubuntu-18.04 Azure VM with a K8 GPU. I highly recommend using at least one GPU (w/ >20GB of memory) with the correct CUDA installed. Make sure the CUDA, Pytorch and Detectron2 have the correct CUDA version if using GPU.

Usage

---

This project contains various scripts to reproduce the results in the paper.

Dataset Preparations

Download and unzip the training and evaluation datasets either locally or in an Azure blob container. For each of the zip datasets, contains the coco annotations .json file and images in the data folder. Start by downloading the OnlyPlanes dataset using the following commands:

sudo chmod u+x scripts/01a_download_OnlyPlanes_dataset.sh
sh scripts/01a_download_OnlyPlanes_dataset.sh

details on the structure of OnlyPlanes Dataset and annotations

The structure of each dataset is as follows:

.
└── OnlyPlanes
    ├── LICENSE.txt
    ├── coco_annotations
    │   ├── all_labels_coco_annotations.json    # all plane labels including fine-grain attributes
    │   ├── binary_plane_coco_annotations.json  # use this for **training on binary planes** 1 class
    │   ├── role_coco_annotations.json          
    │   ├── civil_role_coco_annotations.json    
    │   ├── OnlyPlanes_Role_Counts.csv
    │   └── OnlyPlanes_metadata.csv
    └── images
        ├── 1.png
        └── ...

Train model on Synthetic OnlyPlanes dataset

OnlyPlanes paper provides an object detection and a segmentation model. You can train the model by yourself or directly use the snapshot provided by us. The following script lets you train them using the configurations defined in the OnlyPlanes paper.

To train the dataset run the following, also update the training config .yaml files as necessary (e.g., number of classes)

conda activate OP

TRAIN_IMG_DIR=/path/to/image/directory         #(e.g., datasets/OnlyPlanes/images)
TRAIN_COCO_JSON=/path/to/coco_annotations.json #(e.g., datasets/OnlyPlanes/coco_ds/OnlyPlanes_binary_plane_annotations.json)

VAL_IMG_DIR=/path/to/image/directory         
VAL_COCO_JSON=/path/to/coco_annotations.json 

OUTPUT_FOLDER=/path/to/output_training_results
TRAINCONFIG=/path/to/training_configs.yaml    #(e.g., configs/frcnn_r50_fpn_demo.yaml)
TRAIN_METHOD=bbox #bbox segm (rotated and obb not supported yet)
python train.py --output-folder $OUTPUT_FOLDER --config-file $TRAINCONFIG \
                --train_img_dir $TRAIN_IMG_DIR --train_coco_json $TRAIN_COCO_JSON \
                --val_img_dir $VAL_IMG_DIR --val_coco_json $VAL_COCO_JSON \
                --train_method $TRAIN_METHOD

Evaluation on real images

The inference.py script has been provided to demo the performance of the trained detector or segmentation on a folder containing test images of real world aircrafts.

INPUT_DIR=/path/to/images
OUTPUT_DIR=/path/to/save/predicted_images
CONF=0.5
USE_GPU=True
CKPT_PATH=/path/to/model/model_final.pth
NUM_IMAGES=2

python inference.py --input_dir ${INPUT_DIR} --output_dir ${OUTPUT_DIR} --conf ${CONF} \
                    --use_gpu ${USE_GPU} --ckpt_path ${CKPT_PATH} --num_images ${NUM_IMAGES}

License

The source code of this repository is released only for academic use. See the license file for details.

Notes

The codes of this repository are built upone the folloiwng open sources. Thanks to the authors for sharing the code!

• Pre-trained machine learning models and tools from Detectron2
• Synthetic dataset is generated by Microsoft Mixed Reality

---

Citation

If you find this project useful or utilize the OnlyPlanes Dataset for your research, please kindly cite our paper.

@article{hale2022OnlyPlanes,
  title={OnlyPlanes: Incrementally Tuning Synthetic Training Datasets for Satellite Object Detection},
  author={Hale, Phillip and Tiday, Luke and Urbina, Pedro},
  number={arXiv:TBD},
  year={2022}
}