[1pt] Update CatFIM site comparison tool to create multiple comparisons and more descriptive output CSVs

[EmilyDeardorff]

In this PR, I’ve re-written the catfim_sites_compare.py tool. The updated version can handle more model inputs (including outputs from both flow-based and stage-based CatFIM) and produces additional compiled CSVs for analysis.

CatFIM Sites Compare Outputs:

• Number of outputs depends on how many CatFIM results are provided.
• For example, if 3 versions of flow-based CatFIM are provided, the following outputs will be created:
  • flow_based_compare_all_versions.csv
  • flow_based_<version_1>vs<version_2>.csv
  • flow_based_<version_2>vs<version_3>.csv
• If both flow- and stage-based CatFIM are provided, a separate "compare_all_versions" CSV will be created for each product.

Changes

• tools/catfim/catfim_sites_compare.py:
  • Changed outputs. Instead of one output CSV comparing the two versions provided, the tool will provide one CSV with the compiled statuses from all flow-based outputs, one for stage-based, and then a comparison CSV with before/after status changes for all sequential pairs of CatFIM outputs provided. A log file is no longer being saved.
  • Changed input structure. Instead of specifying -p for previous CatFIM outputs and -n for new ones, a space-delimited list of CatFIM output paths are provided for -p.
  • New optional argument -k specifies to keep only the sites where there has been a status change in the version comparison files.
  • Changed input allowance. User can now input as many CatFIM output folders as they want (and can combine stage- and flow-based outputs).

Testing

I ran the following test command within a docker image and the code performed as expected.

python /foss_fim/tools/catfim/catfim_sites_compare.py -p  '/data/catfim/hand_4_5_11_1_stage_based/ /data/catfim/fim_4_5_2_11_stage_based/ /data/catfim/fim_4_4_0_0_stage_based/ /data/catfim/hand_4_5_11_1_flow_based/ /data/catfim/fim_4_5_2_11_flow_based/ /data/catfim/fim_4_5_2_0_flow_based/' -o '/home/emily.deardorff/notebooks/'

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

Figures

Comparison file example -> comparison of two different sequential versions

All versions file example -> comparison of all versions (for flow- or stage-based)

Explanation of status descriptions:

	Currently Excluded from Processing (so by default unmapped)	Currently Unmapped (but included)	Currently Mapped
Previously Excluded from Processing (so by default unmapped)	NA (these sites don’t make it into the table)	Status Change (Previously excluded, now included but unmapped)	Added (Site where CatFIM was not processed previously but now has mapped CatFIM)
Previously Unmapped (but included)	Status Change (Previously unmapped, now excluded from processing)	No Change (Doesn’t have mapped CatFIM in either version)	Added (Site where CatFIM was not mapped previously but now has mapped CatFIM)
Previously Mapped Currently Mapped	Removed (Previously had mapped CatFIM, now site isn’t being processed)	Removed (Previously had mapped CatFIM, now site isn’t being mapped)	No Change (Has mapped CatFIM in both versions)