V00.01.00003 release (#356)

* cleaning up files which are no longer needed

* fixes after removing forking workflow (#322)

* PR to resolve merge issues

* updated main build as well

* added ability to read in git branch name directly

* manually updated the other files

* fixed number of classes for main build tests (#327)

* fixed number of classes for main build tests

* corrected DATASET.ROOT in builds

* added dev build script

* Fixes for development inside the docker container (#335)

* Fix the mound command for the HRNet pretrained model in the docker readme

* Properly catch InvalidGitRepository exception

* make repo paths consistent with non-docker runs -- this way configs paths do not need to be changed

* Properly catch InvalidGitRepository exception in train.py

* Readme update (#337)

* README updates

* Removing user specific path from config

Authored-by: Fatemeh Zamanian <[email protected]>

* Fixing #324 and #325 (#338)

* update colormap to a non-discrete one -- fixes #324

* fix mask_to_disk to normalize by n_classes

* changes to test.py

* Updating data.py

* bug fix

* increased timeout time for main_build

* retrigger build

* retrigger the build

* increase timeout

* fixes 318 (#339)

* finished 318

* increased checkerboard test timeout

* fix 333 (#340)

* added label correction to train gradient

* changing the gradient data generator to take inline/crossline argument conssistent with the patchloader

* changing variable name to be more descriptive


Co-authored-by: maxkazmsft <[email protected]>

* bug fix to model predictions (#345)

* replace hrnet with seresnet in experiments - provides stable default model (#343)

Co-authored-by: yalaudah <[email protected]>
Co-authored-by: Fatemeh <[email protected]>

Author: 3 people

README.md

@@ -3,12 +3,13 @@
 
 This repository shows you how to perform seismic imaging and interpretation on Azure. It empowers geophysicists and data scientists to run seismic experiments using state-of-art DSL-based PDE solvers and segmentation algorithms on Azure.  
 
+
 The repository provides sample notebooks, data loaders for seismic data, utilities, and out-of-the-box ML pipelines, organized as follows:
 - **sample notebooks**: these can be found in the `examples` folder - they are standard Jupyter notebooks which highlight how to use the codebase by walking the user through a set of pre-made examples
 - **experiments**: the goal is to provide runnable Python scripts that train and test (score) our machine learning models in the `experiments` folder. The models themselves are swappable, meaning a single train script can be used to run a different model on the same dataset by simply swapping out the configuration file which defines the model. 
-- **pip installable utilities**: we provide `cv_lib` and `deepseismic_interpretation` utilities (more info below) which are used by both sample notebooks and experiments mentioned above
+- **pip installable utilities**: we provide `cv_lib` and `interpretation` utilities (more info below) which are used by both sample notebooks and experiments mentioned above
 
-DeepSeismic currently focuses on Seismic Interpretation (3D segmentation aka facies classification) with experimental code provided around Seismic Imaging in the contrib folder.
+DeepSeismic currently focuses on Seismic Interpretation (mainly facies classification) with experimental code provided around Seismic Imaging in the contrib folder.
 
 ### Quick Start
 
@@ -26,7 +27,7 @@ If you run into any problems, chances are your problem has already been solved i
 The notebook is designed to be run in demo mode by default using a pre-trained model in under 5 minutes on any reasonable Deep Learning GPU such as nVidia K80/P40/P100/V100/TitanV.
 
 ### Azure Machine Learning
-[Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/) enables you to train and deploy your machine learning models and pipelines at scale, ane leverage open-source Python frameworks, such as PyTorch, TensorFlow, and scikit-learn. If you are looking at getting started with using the code in this repository with Azure Machine Learning, refer to [Azure Machine Learning How-to](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml) to get started.
+[Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/) enables you to train and deploy your machine learning models and pipelines at scale, and leverage open-source Python frameworks, such as PyTorch, TensorFlow, and scikit-learn. If you are looking at getting started with using the code in this repository with Azure Machine Learning, refer to [Azure Machine Learning How-to](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml) to get started.
 
 ## Interpretation
 For seismic interpretation, the repository consists of extensible machine learning pipelines, that shows how you can leverage state-of-the-art segmentation algorithms (UNet, SEResNET, HRNet) for seismic interpretation.
@@ -120,9 +121,12 @@ To prepare the data for the experiments (e.g. split into train/val/test), please
 cd scripts
 
 # For patch-based experiments
-python prepare_dutchf3.py split_train_val patch --data_dir=${data_dir} --label_file=train/train_labels.npy --output_dir=splits \
+python prepare_dutchf3.py split_train_val patch --data_dir=${data_dir}/data --label_file=train/train_labels.npy --output_dir=splits \
 --stride=50 --patch_size=100 --split_direction=both
 
+# For section-based experiments
+python prepare_dutchf3.py split_train_val section --data-dir=${data_dir}/data --label_file=train/train_labels.npy --output_dir=splits \ --split_direction=both
+
 # go back to repo root
 cd ..
 ```
@@ -164,7 +168,7 @@ We use [YACS](https://github.com/rbgirshick/yacs) configuration library to manag
 - __yml config files__ - YAML configuration files under `configs/` are typically created one for each experiment. These are meant to be used for repeatable experiment runs and reproducible settings. Each configuration file only overrides the options that are changing in that experiment (e.g. options loaded from `defaults.py` during an experiment run will be overridden by arguments loaded from the yaml file). As an example, to use yml configuration file with the training script, run:
 
     ```
-    python train.py --cfg "configs/hrnet.yaml"
+    python train.py --cfg "configs/seresnet_unet.yaml"
     ```
 
 - __command line__ - Finally, options can be passed in through `options` argument, and those will override arguments loaded from the configuration file. We created CLIs for all our scripts (using Python Fire library), so you can pass these options via command-line arguments, like so:
@@ -229,8 +233,8 @@ This section contains benchmarks of different algorithms for seismic interpretat
 
 
 #### Reproduce benchmarks
-In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments/dutchf3_patch/local/train.sh))
-which will run the training for any configuration you pass in. Once you have run the training you will need to run the [test.sh](experiments/dutchf3_patch/local/test.sh) script. Make sure you specify
+In order to reproduce the benchmarks, you will need to navigate to the [experiments](experiments) folder. In there, each of the experiments are split into different folders. To run the Netherlands F3 experiment navigate to the [dutchf3_patch/local](experiments/interpretation/dutchf3_patch/local) folder. In there is a training script [([train.sh](experiments/interpretation/dutchf3_patch/local/train.sh))
+which will run the training for any configuration you pass in. Once you have run the training you will need to run the [test.sh](experiments/interpretation/dutchf3_patch/local/test.sh) script. Make sure you specify
 the path to the best performing model from your training run, either by passing it in as an argument or altering the YACS config file. 
 
 ## Contributing

contrib/experiments/interpretation/dutchf3_patch/distributed/train.py

@@ -2,10 +2,10 @@
 # Licensed under the MIT License.
 #
 # To Run on 2 GPUs
-# python -m torch.distributed.launch --nproc_per_node=2 train.py --cfg "configs/hrnet.yaml"
+# python -m torch.distributed.launch --nproc_per_node=2 train.py --cfg "configs/seresnet_unet.yaml"
 #
 # To Test:
-# python -m torch.distributed.launch --nproc_per_node=2 train.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/hrnet.yaml" --debug
+# python -m torch.distributed.launch --nproc_per_node=2 train.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/seresnet_unet.yaml" --debug
 #
 # /* spell-checker: disable */
 """Train models on Dutch F3 dataset
@@ -138,7 +138,7 @@ def run(*options, cfg=None, local_rank=0, debug=False):
         stride=config.TRAIN.STRIDE,
         patch_size=config.TRAIN.PATCH_SIZE,
         augmentations=train_aug,
-    )    
+    )
 
     val_set = TrainPatchLoader(
         config.DATASET.ROOT,
@@ -154,10 +154,10 @@ def run(*options, cfg=None, local_rank=0, debug=False):
 
     if debug:
         val_set = data.Subset(val_set, range(config.VALIDATION.BATCH_SIZE_PER_GPU))
-        train_set = data.Subset(train_set, range(config.TRAIN.BATCH_SIZE_PER_GPU*2))
-    
+        train_set = data.Subset(train_set, range(config.TRAIN.BATCH_SIZE_PER_GPU * 2))
+
     logger.info(f"Training examples {len(train_set)}")
-    logger.info(f"Validation examples {len(val_set)}")    
+    logger.info(f"Validation examples {len(val_set)}")
 
     train_sampler = torch.utils.data.distributed.DistributedSampler(train_set, num_replicas=world_size, rank=local_rank)
 
@@ -193,7 +193,7 @@ def run(*options, cfg=None, local_rank=0, debug=False):
 
     model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device], find_unused_parameters=True)
 
-    snapshot_duration = epochs_per_cycle * len(train_loader) if not debug else 2*len(train_loader)
+    snapshot_duration = epochs_per_cycle * len(train_loader) if not debug else 2 * len(train_loader)
 
     warmup_duration = 5 * len(train_loader)
 

contrib/experiments/interpretation/dutchf3_patch/distributed/train.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 export PYTHONPATH=/data/home/mat/repos/DeepSeismic/interpretation:$PYTHONPATH
-python -m torch.distributed.launch --nproc_per_node=8 train.py --cfg configs/hrnet.yaml
+python -m torch.distributed.launch --nproc_per_node=8 train.py --cfg configs/seresnet_unet.yaml

contrib/experiments/interpretation/dutchf3_voxel/README.md

@@ -1,13 +1,13 @@
-First, make sure that `${HOME}/data/dutch_f3` folder exists and you have write access.
+First, make sure that `${HOME}/data/dutch` folder exists and you have write access.
 
 Next, to get the main input dataset which is the [Dutch F3 dataset](https://terranubis.com/datainfo/Netherlands-Offshore-F3-Block-Complete), 
 navigate to [MalenoV](https://github.com/bolgebrygg/MalenoV) project website and follow the links (which will lead to 
 [this](https://drive.google.com/drive/folders/0B7brcf-eGK8CbGhBdmZoUnhiTWs) download). Save this file as 
-`${HOME}/data/dutch_f3/data.segy`
+`${HOME}/data/dutch/data.segy`
 
 To download the train and validation masks, from the root of the repo, run
 ```bash
-./contrib/scripts/get_F3_voxel.sh ${HOME}/data/dutch_f3
+./contrib/scripts/get_F3_voxel.sh ${HOME}/data/dutch
 ```
 
 This will also download train and validation masks to the same location as data.segy.

contrib/experiments/interpretation/dutchf3_voxel/configs/texture_net.yaml

@@ -15,7 +15,7 @@ WINDOW_SIZE: 65
 
 DATASET:
   NUM_CLASSES: 2
-  ROOT: /home/maxkaz/data/dutchf3
+  ROOT: /home/username/data/dutchf3
   FILENAME: data.segy  
 
 MODEL:

contrib/experiments/interpretation/dutchf3_voxel/train.py

@@ -158,7 +158,7 @@ def run(*options, cfg=None):
     def _select_pred_and_mask(model_out):
         # receive a tuple of (x, y_pred), y
         # so actually in line 51 of
-        # cv_lib/cv_lib/segmentation/dutch_f3/metrics/__init__.py
+        # cv_lib/cv_lib/segmentation/dutch/metrics/__init__.py
         # we do the following line, so here we just select the model
         # _, y_pred = torch.max(model_out[0].squeeze(), 1, keepdim=True)
         y_pred = model_out[0].squeeze()

contrib/experiments/interpretation/penobscot/local/test.py

@@ -2,7 +2,7 @@
 # Licensed under the MIT License.
 #
 # To Test:
-# python test.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/hrnet.yaml" --debug
+# python test.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/seresnet_unet.yaml" --debug
 #
 # /* spell-checker: disable */
 """Train models on Penobscot dataset
@@ -244,9 +244,7 @@ def _select_max(pred_tensor):
     def _tensor_to_numpy(pred_tensor):
         return pred_tensor.squeeze().cpu().numpy()
 
-    transform_func = compose(
-        np_to_tb, decode_segmap, _tensor_to_numpy,
-    )
+    transform_func = compose(np_to_tb, decode_segmap, _tensor_to_numpy,)
 
     transform_pred = compose(transform_func, _select_max)
 

contrib/experiments/interpretation/penobscot/local/train.py

@@ -2,7 +2,7 @@
 # Licensed under the MIT License.
 #
 # To Test:
-# python train.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/hrnet.yaml" --debug
+# python train.py TRAIN.END_EPOCH 1 TRAIN.SNAPSHOTS 1 --cfg "configs/seresnet_unet.yaml" --debug
 #
 # /* spell-checker: disable */
 """Train models on Penobscot dataset
@@ -43,6 +43,7 @@
 
 mask_value = 255
 
+
 def _prepare_batch(batch, device=None, non_blocking=False):
     x, y, ids, patch_locations = batch
     return (
@@ -253,9 +254,7 @@ def _select_max(pred_tensor):
     def _tensor_to_numpy(pred_tensor):
         return pred_tensor.squeeze().cpu().numpy()
 
-    transform_func = compose(
-        np_to_tb, decode_segmap, _tensor_to_numpy,
-    )
+    transform_func = compose(np_to_tb, decode_segmap, _tensor_to_numpy,)
 
     transform_pred = compose(transform_func, _select_max)
 

contrib/experiments/interpretation/voxel2pixel/test_parallel.py

@@ -245,7 +245,7 @@ def main_worker(gpu, ngpus_per_node, args):
 
 
 parser = argparse.ArgumentParser(description="Seismic Distributed Scoring")
-parser.add_argument("-d", "--data", default="/home/maxkaz/data/dutchf3", type=str, help="default dataset folder name")
+parser.add_argument("-d", "--data", default="/home/username/data/dutchf3", type=str, help="default dataset folder name")
 parser.add_argument(
     "-s",
     "--slice",

contrib/experiments/interpretation/voxel2pixel/train.py

@@ -15,7 +15,7 @@
 import utils
 
 # Parameters
-ROOT_PATH = "/home/maxkaz/data/dutchf3"
+ROOT_PATH = "/home/username/data/dutchf3"
 INPUT_VOXEL = "data.segy"
 TRAIN_MASK = "inline_339.png"
 VAL_MASK = "inline_405.png"