From 99f16010cb36e245f005fda93d8b029ad3f21203 Mon Sep 17 00:00:00 2001
From: Rob Hanna - NOAA <90854818+RobHanna-NOAA@users.noreply.github.com>
Date: Fri, 23 Dec 2022 14:03:40 -0600
Subject: [PATCH] Make GMS HAND post-processing independent
---
docs/CHANGELOG.md | 54 +++++-
gms_pipeline.sh | 52 ++---
gms_run_branch.sh | 159 +++------------
gms_run_post_processing.sh | 182 ++++++++++++++++++
gms_run_unit.sh | 75 ++------
src/bash_functions.env | 10 +
src/filter_catchments_and_add_attributes.py | 22 +--
src/gms/delineate_hydros_and_produce_HAND.sh | 4 +-
src/gms/derive_level_paths.py | 56 +++---
src/gms/run_by_unit.sh | 19 +-
src/split_flows.py | 25 +--
src/usgs_gage_unit_setup.py | 9 +-
unit_tests/gms/derive_level_paths_params.json | 6 +-
.../gms/derive_level_paths_unittests.py | 94 +--------
14 files changed, 354 insertions(+), 413 deletions(-)
create mode 100755 gms_run_post_processing.sh
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 401a91b75..90abdebc9 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -1,6 +1,56 @@
All notable changes to this project will be documented in this file.
We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
+## v4.0.16.0 - 2022-12-20 - [PR #768](https://github.com/NOAA-OWP/inundation-mapping/pull/768)
+
+`gms_run_branch.sh` was processing all of the branches iteratively, then continuing on to a large post processing portion of code. That has now be split to two files, one for branch iteration and the other file for just post processing.
+
+Other minor changes include:
+- Removing the system where a user could override `DropStreamOrders` where they could process streams with stream orders 1 and 2 independently like other GMS branches. This option is now removed, so it will only allow stream orders 3 and higher as gms branches and SO 1 and 2 will always be in branch zero.
+
+- The `retry` flag on the three gms*.sh files has been removed. It did not work correctly and was not being used. Usage of it would have created unreliable results.
+
+### Additions
+
+- `gms_run_post_processing.sh`
+ - handles all tasks from after `gms_run_branch.sh` to this file, except for output cleanup, which stayed in `gms_run_branch.sh`.
+ - Can be run completely independent from `gms_run_unit.sh` or gms_run_branch.sh` as long as all of the files are in place. And can be re-run if desired.
+
+### Changes
+
+- `gms_pipeline.sh`
+ - Remove "retry" system.
+ - Remove "dropLowStreamOrders" system.
+ - Updated for newer reusable output date/time/duration system.
+ - Add call to new `gms_run_post_processing.sh` file.
+
+- `gms_run_branch.sh`
+ - Remove "retry" system.
+ - Remove "dropLowStreamOrders" system.
+ - Updated for newer reusable output date/time/duration system.
+ - Removed most code from below the branch iterator to the new `gms_run_post_processing.sh` file. However, it did keep the branch files output cleanup and non-zero exit code checking.
+
+- `gms_run_unit.sh`
+ - Remove "retry" system.
+ - Remove "dropLowStreamOrders" system.
+ - Updated for newer reusable output date/time/duration system.
+
+- `src`
+ - `bash_functions.env`: Added a new method to make it easier / simpler to calculation and display duration time.
+ - `filter_catchments_and_add_attributes.py`: Remove "dropLowStreamOrders" system.
+ - `split_flows.py`: Remove "dropLowStreamOrders" system.
+ - `usgs_gage_unit_setup.py`: Remove "dropLowStreamOrders" system.
+
+- `gms`
+ - `delineate_hydros_and_produced_HAND.sh` : Remove "dropLowStreamOrders" system.
+ - `derive_level_paths.py`: Remove "dropLowStreamOrders" system and some small style updates.
+ - `run_by_unit.sh`: Remove "dropLowStreamOrders" system.
+
+- `unit_tests/gms`
+ - `derive_level_paths_params.json` and `derive_level_paths_unittests.py`: Remove "dropLowStreamOrders" system.
+
+
+
## v4.0.15.0 - 2022-12-20 - [PR #758](https://github.com/NOAA-OWP/inundation-mapping/pull/758)
This merge addresses feedback received from field users regarding CatFIM. Users wanted a Stage-Based version of CatFIM, they wanted maps created for multiple intervals between flood categories, and they wanted documentation as to why many sites are absent from the Stage-Based CatFIM service. This merge seeks to address this feedback. CatFIM will continue to evolve with more feedback over time.
@@ -80,13 +130,11 @@ Fixes inundation of nodata areas of REM.
- `tools/inundation.py`: Assigns depth a value of `0` if REM is less than `0`
-
-
## v4.0.13.1 - 2022-12-09 - [PR #743](https://github.com/NOAA-OWP/inundation-mapping/pull/743)
This merge adds the tools required to generate Alpha metrics by hydroid. It summarizes the Apha metrics by branch 0 catchment for use in the Hydrovis "FIM Performance" service.
-## Additions
+### Additions
- `pixel_counter.py`: A script to perform zonal statistics against raster data and geometries
- `pixel_counter_functions.py`: Supporting functions
diff --git a/gms_pipeline.sh b/gms_pipeline.sh
index 4aa9c3216..2907ce04e 100755
--- a/gms_pipeline.sh
+++ b/gms_pipeline.sh
@@ -4,10 +4,11 @@ usage ()
{
echo
echo 'Produce GMS hydrofabric datasets for unit and branch scale.'
- echo 'Usage : gms_pipeline.sh [REQ: -u - -n ]'
- echo ' [OPT: -h -c -j ] -o -r'
- echo ' -ud -bd '
- echo ' -zd -a