Initial Commit

Author: D-X-Y

.gitignore

@@ -0,0 +1,14 @@
+__pycache__
+*/__pycache__
+*/*/__pycache__
+snapshots
+cache_data/cache
+cache_data/lists
+.DS_Store
+*/.DS_Store
+*/*/.DS_Store
+*.swp
+*/*.swp
+*/*/*.swp
+*/AFLWinfo_release.mat
+AFLWinfo_release.mat

CODE_OF_CONDUCT.md

@@ -0,0 +1,3 @@
+# Code of Conduct
+
+Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct) so that you can understand what actions will and will not be tolerated.

CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing to Supervision-by-Registration (SBR)
+We want to make contributions to this project as easy and transparent as possible.
+
+## Our Development Process
+Preliminary Implementations.
+
+## Pull Requests
+	We actively welcome your pull requests.
+
+	1. Fork the repo and create your branch from `master`.
+	2. If you've added code that should be tested, add tests.
+	3. If you've changed APIs, update the documentation.
+	4. Ensure the test suite passes.
+	5. Make sure your code lints.
+	6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+	In order to accept your pull request, we need you to submit a CLA. You only need
+	to do this once to work on any of Facebook's open source projects.
+
+	Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+	We use GitHub issues to track public bugs. Please ensure your description is
+	clear and has sufficient instructions to be able to reproduce the issue.
+
+	Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+	disclosure of security bugs. In those cases, please go through the process
+	outlined on that page and do not file a public issue.
+
+## Coding Style  
+	* 2 spaces for indentation rather than tabs
+	* ...
+
+## License
+	By contributing to SBR, you agree that your contributions will be licensed
+	under the LICENSE file in the root directory of this source tree.

LICENSE

@@ -0,0 +1,120 @@
+# Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors
+By Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, Yaser Sheikh
+
+University of Technology Sydney, Facebook Reality Labs
+
+## Introduction
+We propose a method to find facial landmarks (e.g. corner of eyes, corner of mouth, tip of nose, etc) more precisely.
+Our method utilizes the fact that objects move smoothly in a video sequence (i.e. optical flow registration) to improve an existing facial landmark detector.
+The key novelty is that no additional human annotations are necessary to improve the detector, hence it is an “unsupervised approach”.
+
+![demo](https://github.com/facebookresearch/supervision-by-registration/blob/master/cache_data/cache/demo.gif)
+
+## Citation
+If you find that Supervision-by-Registration helps your research, please cite the paper:
+```
+@inproceedings{dong2018sbr,
+  title={{Supervision-by-Registration}: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors},
+  author={Dong, Xuanyi and Yu, Shoou-I and Weng, Xinshuo and Wei, Shih-En and Yang, Yi and Sheikh, Yaser},
+  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
+  pages={360--368},
+  year={2018}
+}
+```
+
+## Requirements
+- PyTorch >= 0.4.0
+- Python3.6
+
+## Data Preparation
+
+See the README in `cache_data`.
+
+### Dataset Format
+Each dataset is saved as one file, in which each row indicates one specific face in one image or one video frame.
+The format of one line : 
+```
+image_path annotation_path x1 y1 x2 y2 (face_size)
+```
+- *image_path*: the image (video frame) file path of that face.
+- *annotation_path*: the annotation file path of that face (annotation is the coordinates of all landmarks)
+- *x1, y1, x2, y2*: the coordinates of left-upper and right-lower points of the face bounding box.
+- *face_size*: an optional item. If set this value, we use the `face_size` to compute the NME; otherwise, we use the distance between two pre-defined points to compute the NME.
+
+## Training
+
+See the `configs` directory for some example configurations.
+### Basic Training
+```
+python ./exps/basic_main.py [<required arguments>]
+```
+The argument list is loaded by `./lib/config_utils/basic_args.py`.
+An examples script can is `./scripts/300W-DET.sh`, and you can simple run to train the base detector on the `300-W` dataset.
+```
+sh scripts/300W-DET.sh
+```
+
+### Improving the Detector by SBR
+```
+python ./exps/lk_main.py [<required arguments>]
+```
+The argument list is loaded by `./lib/config_utils/lk_args.py`.
+
+
+#### An example to train SBR on the unlabeled sequences
+The `init_model` parameter is the path to the detector trained in the `Basic Training` section.
+```
+sh scripts/demo_sbr.sh
+```
+To see visualization results use the commands in `Visualization`.
+
+#### An example to train SBR on your own data
+See the script `./scripts/sbr_example.sh`, and some parameters should be replaced by your own data.
+
+
+## Evaluation
+
+When using the `basic_main.py` or `lk_main.py`, we evaluate the testing datasets automatically.
+
+To evaluate a single image, you can use the following script to compute the coordinates of 68 facial landmarks of the target image:
+```
+python ./exps/eval.py --image ./cache_data/cache/self.jpeg --model ./snapshots/300W-CPM-DET/checkpoint/cpm_vgg16-epoch-049-050.pth --face 250 150 900 1100 --save ./cache_data/cache/test.jpeg
+```
+- image : the input image path
+- model : the snapshot path
+- face  : the face bounding box
+- save  : save the visualized results
+
+
+## Visualization
+
+After training the SBR on the demo video or models on other datasets, you can use the `./exps/vis.py` code to generate the visualization results.
+```
+python ./exps/vis.py --meta snapshots/CPM-SBR/metas/eval-start-eval-00-01.pth --save cache_data/cache/demo-detsbr-vis
+ffmpeg -start_number 3 -i cache_data/cache/demo-detsbr-vis/image%04d.png -b:v 30000k -vf "fps=30" -pix_fmt yuv420p cache_data/cache/demo-detsbr-vis.mp4
+
+python ./exps/vis.py --meta snapshots/CPM-SBR/metas/eval-epoch-049-050-00-01.pth --save cache_data/cache/demo-sbr-vis
+ffmpeg -start_number 3 -i cache_data/cache/demo-sbr-vis/image%04d.png -b:v 30000k -vf "fps=30" -pix_fmt yuv420p cache_data/cache/demo-sbr-vis.mp4
+```
+- meta : the saved prediction files
+- save : the directory path to save the visualization results
+
+
+## License
+supervision-by-registration is released under the [CC-BY-NC license](https://github.com/facebookresearch/supervision-by-registration/blob/master/LICENSE).
+
+
+## Useful information
+
+### 1. train on your own video data
+You should look at the `./lib/datasets/VideoDataset.py` and `./lib/datasets/parse_utils.py`, and add how to find the neighbour frames when giving one image path.
+For more details, see the `parse_basic` function in `lib/datasets/parse_utils.py`.
+
+### 2. warnings when training the AFLW datase
+It is ok to show the following warnings. Since some images in the AFLW dataset are in the wrong format, PIL will raise some warnings when loading these images. These warnings do not affect the training performance.
+```
+TiffImagePlugin.py:756: UserWarning: Corrupt EXIF data.  Expecting to read 12 bytes but only got 6.
+```
+
+### Contact
+To ask questions or report issues, please open an issue on [the issues tracker](https://github.com/facebookresearch/supervision-by-registration/issues).

README.md

@@ -0,0 +1,3 @@
+*.lst
+temp
+EX300.sh

cache_data/.gitignore

@@ -0,0 +1,147 @@
+# Dataset Preparation
+The raw dataset should be put into the `$HOME/datasets/landmark-datasets`. The layout should be organized as the following screen shot.
+
+![layout](https://github.com/facebookresearch/supervision-by-registration/blob/master/cache_data/cache/dir-layout.png)
+
+## [300-W](https://ibug.doc.ic.ac.uk/resources/300-W/)
+
+### Download
+- 300-W consits of several different datasets
+- Create directory to save images and annotations: mkdir ~/datasets/landmark-datasets/300W
+- To download i-bug: https://ibug.doc.ic.ac.uk/download/annotations/ibug.zip
+- To download afw: https://ibug.doc.ic.ac.uk/download/annotations/afw.zip
+- To download helen: https://ibug.doc.ic.ac.uk/download/annotations/helen.zip
+- To download lfpw: https://ibug.doc.ic.ac.uk/download/annotations/lfpw.zip
+- To download the bounding box annotations: https://ibug.doc.ic.ac.uk/media/uploads/competitions/bounding_boxes.zip
+- In the folder of `~/datasets/landmark-datasets/300W`, there are four zip files ibug.zip, afw.zip, helen.zip, and lfpw.zip
+```
+unzip ibug.zip -d ibug
+mv ibug/image_092\ _01.jpg ibug/image_092_01.jpg
+mv ibug/image_092\ _01.pts ibug/image_092_01.pts
+
+unzip afw.zip -d afw
+unzip helen.zip -d helen
+unzip lfpw.zip -d lfpw
+unzip bounding_boxes.zip ; mv Bounding\ Boxes Bounding_Boxes
+```
+The 300W directory is in `$HOME/datasets/landmark-datasets/300W` and the sturecture is:
+```
+-- afw
+-- Bounding_boxes
+-- helen
+-- ibug
+-- lfpw
+```
+
+Then you use the script to generate the 300-W list files.
+```
+python generate_300W.py
+```
+All list files will be saved into `./lists/300W/`. The files `*.DET` use the face detecter results for face bounding box. `*.GTB` use the ground-truth results for face bounding box.
+
+#### can not find the `*.mat` files for 300-W.
+The download link is in the official [300-W website](https://ibug.doc.ic.ac.uk/resources/300-W).
+```
+https://ibug.doc.ic.ac.uk/media/uploads/competitions/bounding_boxes.zip
+```
+The zip file should be unzipped, and all extracted mat files should be put into `$HOME/datasets/landmark-datasets/300W/Bounding_Boxes`.
+
+## [AFLW](https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/)
+
+Download the aflw.tar.gz file in `$HOME/datasets/landmark-datasets` and extract it by `tar xzvf aflw.tar.gz`.
+```
+mkdir $HOME/datasets/landmark-datasets/AFLW
+cp -r aflw/data/flickr $HOME/datasets/landmark-datasets/AFLW/images
+```
+
+The structure of AFLW is:
+```
+--images
+  --0
+  --2
+  --3
+```
+
+Download the [AFLWinfo_release.mat](http://mmlab.ie.cuhk.edu.hk/projects/compositional/AFLWinfo_release.mat) from [this website](http://mmlab.ie.cuhk.edu.hk/projects/compositional.html) into `./cache_data`. This is the revised annotation of the full AFLW dataset.
+
+Generate the AFLW dataset list file into `./lists/AFLW`.
+```
+python aflw_from_mat.py
+```
+
+## [300VW](https://ibug.doc.ic.ac.uk/resources/300-VW/)
+Download `300VW_Dataset_2015_12_14.zip` into `$HOME/datasets/landmark-datasets` and unzip it into `$HOME/datasets/landmark-datasets/300VW_Dataset_2015_12_14`.
+
+Use the following command to extract the raw video into the image format.
+```
+python extrct_300VW.py
+sh ./cache/Extract300VW.sh
+```
+
+Generate the 300-VW dataset list file.
+```
+python generate_300VW.py
+```
+
+## a short demo video sequence
+
+The raw video is `./cache_data/cache/demo-sbr.mp4`.
+- use `ffmpeg -i ./cache/demo-sbr.mp4 ./cache/demo-sbrs/image%04d.png` to extract the frames into `/cache/demo-sbrs/`
+Then use `python demo_list.py` to generate the list file for the demo video.
+
+# Citation
+If you use the 300-W dataset, please cite the following papers.
+```
+@article{sagonas2016300,
+  title={300 faces in-the-wild challenge: Database and results},
+  author={Sagonas, Christos and Antonakos, Epameinondas and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja},
+  journal={Image and Vision Computing},
+  volume={47},
+  pages={3--18},
+  year={2016},
+  publisher={Elsevier}
+}
+@inproceedings{sagonas2013300,
+  title={300 faces in-the-wild challenge: The first facial landmark localization challenge},
+  author={Sagonas, Christos and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja},
+  booktitle={Proceedings of the IEEE International Conference on Computer Vision Workshops},
+  pages={397--403},
+  year={2013},
+  organization={IEEE}
+}
+```
+If you use the 300-VW dataset, please cite the following papers.
+```
+@inproceedings{chrysos2015offline,
+  title={Offline deformable face tracking in arbitrary videos},
+  author={Chrysos, Grigoris G and Antonakos, Epameinondas and Zafeiriou, Stefanos and Snape, Patrick},
+  booktitle={Proceedings of the IEEE International Conference on Computer Vision Workshops},
+  pages={1--9},
+  year={2015}
+}
+@inproceedings{shen2015first,
+  title={The first facial landmark tracking in-the-wild challenge: Benchmark and results},
+  author={Shen, Jie and Zafeiriou, Stefanos and Chrysos, Grigoris G and Kossaifi, Jean and Tzimiropoulos, Georgios and Pantic, Maja},
+  booktitle={Proceedings of the IEEE International Conference on Computer Vision Workshops},
+  pages={50--58},
+  year={2015}
+}
+@inproceedings{tzimiropoulos2015project,
+  title={Project-out cascaded regression with an application to face alignment},
+  author={Tzimiropoulos, Georgios},
+  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
+  pages={3659--3667},
+  year={2015}
+}
+```
+If you use the AFLW dataset, please cite the following papers.
+```
+@inproceedings{koestinger2011annotated,
+  title={Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization},
+  author={Koestinger, Martin and Wohlhart, Paul and Roth, Peter M and Bischof, Horst},
+  booktitle={IEEE International Conference on Computer Vision Workshops},
+  pages={2144--2151},
+  year={2011},
+  organization={IEEE}
+}
+```