Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exposure filtering script #85

Closed
wants to merge 34 commits into from
Closed

Exposure filtering script #85

wants to merge 34 commits into from

Conversation

erialC-P
Copy link
Collaborator

@erialC-P erialC-P commented May 7, 2024

This PR includes significant changes to the exposure calculation and custom filtering workflow, based on really helpful review and feedback from the first version.

Major changes to the workflow include the following:

  • Restructuring of the exposure.py script, including breaking out sections of the original function into smaller functions and reducing repetition in calculations
  • Removed pixel based calculations for all 'spatial and temporal filter' calculations of exposure (calculating exposure using the default (filters=None) settings continues to perform a pixel based calculation for the entirety of the input epoch). This workflow takes the input area of interest and calculates a single high temporal/low spatial resolution tidal model for the central coordinates and calculates exposure % against this. This approach to the exposure calculation is designed specifically for sandbox calculations where big area's of interest are not supported. Consequently, the speed of the exposure calculation in the DEA Sandbox is significantly reduced compared to previous iterations of the workflow.
  • Removed timezone conversion to local dates/times in the day/night calculations of exposure, transitioning to UTC time. Testing against local sunrise and sunset times confirmed that UTC was replicating the local dates/times.
  • Confirmation that peak detection routines used to identify hightide and lowtide datetimes performed as expected across diurnal, semi-diurnal and mixed tidal regimes
  • Finally, a major restructure of the calculation. Unless 'unfiltered' is called (whereby a pixel-based tide model is produced), a single high temporal low spatial resolution tide model is generated, irrespective of how many filters are called. All filter datetimes and tide heights are extracted from the tide model. From these, a tideheight quantile curve is generated and compared against dem elevation per pixel whereby the quantile % is equivalent to the exposure time %.

Incorporation of recommendations from the initial review of this workflow have helped to:

  • simplify the structure of the workflow
  • streamline the calculations
  • significantly speed up the computation

Spatial and Temporal functions isolated from main exposure function call.
Reduced modelling to ds scale and removing pixel based calculations from filtered workflows. I think we should keep all-time exposure calculation as pixel based to match the continental workflows.
TODO: change day/night filtering to UTC time and remove regional sampling of datasets.
TODO: test outputs to ensure sensible regional day/night assignments of UTC datetimes, incorporate 'exposure_edited.py' into master 'exposure.py' script, PR changes into main
todo: propagate changes to function call through notebooks and elevation.py
todo: update changes into exposure.py and change func load calls in elevation.py, CLI notebook and workflow notebook
… as intertidal_workflow.ipynb

PR changes document significantly revised exposure.py workflow and simplification of the exposure computation for custom/filtered workflows
remove script and notebook used during exposure filter testing
@erialC-P erialC-P requested review from vnewey and robbibt May 7, 2024 05:46
Copy link

github-actions bot commented May 7, 2024

For full integration test results, refer to the Tests directory README.

@codecov-commenter
Copy link

codecov-commenter commented May 7, 2024

Codecov Report

Attention: Patch coverage is 25.68306% with 136 lines in your changes are missing coverage. Please review.

Project coverage is 57.0%. Comparing base (00fdde3) to head (1177357).

Current head 1177357 differs from pull request most recent head 3a476d4

Please upload reports for the commit 3a476d4 to get more accurate results.

Files Patch % Lines
intertidal/exposure.py 24.4% 136 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main     #85     +/-   ##
=======================================
- Coverage   65.5%   57.0%   -8.6%     
=======================================
  Files          8       8             
  Lines        714     887    +173     
=======================================
+ Hits         468     506     +38     
- Misses       246     381    +135     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

erialC-P and others added 11 commits May 9, 2024 00:41
Todo: fix filtered exposure outputs which are no longer working after updating tidal model calls
…ing. Also tested a fix for calculating high/lowtide exposure in diurnal regimes.

TODO: apply the fix to temporal modelling.
TODO: implement inequality testing on the peak detection to identify diurnal regimes and simplify exposure calculation to use low_peaks2 (or high_peaks2) in these settings accordingly
@erialC-P erialC-P marked this pull request as ready for review May 13, 2024 01:53
Copy link
Member

@robbibt robbibt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @erialC-P, thanks so much for all the changes you've made - this version is super super nice. I've pushed a few additional changes myself, mainly:

  1. Black formatting and added docstrings
  2. Updated all filter options to lowercase_snakes
  3. Refactored the tide modelling code so tide modelling only ever happens once
  4. Updated spatial_filters and temporal_filters to no longer pass externally created objects in and out of the function - now only the outputs are passed out and then assigned into the external objects outside of the functions
  5. Some small changes to variable names and docstrings (e.g. xarray.Datasets were being referred to as dicts in some parts of the code, which I think was probably a legacy of older versions

I think it's pretty much ready to go now as-is, but here's a few other things that I think would be worth focusing a bit of time on if you have capacity:

  1. As discussed on Teams, it would be really nice if both temporal_filters and spatial_filters returned exactly the same output: just a filtered list of timesteps. These outputs could then be handled nicely in one single place outside of those functions using the same consistent code, avoiding duplication and ensuring that exactly the same process/logic is applied to every filter, regardless of its type. I think this could lead to cleaner code and a more consistent user experience overall.
  2. If you wanted to, you could probably think about pulling out some parts of spatial_filters into even smaller sub-functions (e.g. things like the neap/low/high bits where similar code is repeated a few times), and call those subfuncs inside relevant parts of spatial_filters (would make it super easy to update those methods in the future as you could just tweak the mini sub-function
  3. I think there's quite a few places where docstrings haven't been updated for updated code. Would be worth going through the script carefully before we merge just to make sure all the doco is up-to-date.
  4. I don't personally like the name "spatial" in spatial_filters. I think it's potentially going to add confusion as it has such geographic connotations... would just something like "tidal_filters" be beter? Or "tidalstage_filters"?
  5. Also as discussed, it could be neat if we could do away with combined_filters and instead just use filters, perhaps letting users pass in tuples of filters inside the list. But this is very much a "nice to have" and not a major priority. 🙂

@robbibt robbibt added the enhancement New feature or request label May 24, 2024
@erialC-P erialC-P closed this Aug 5, 2024
erialC-P added a commit that referenced this pull request Aug 6, 2024
Removed a outstanding references to spatial filtering and ensured script is up to date and consistent with closed PR #85
@robbibt robbibt deleted the exposure_filters branch September 24, 2024 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants