Notebook runtime improvements #1245
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JM-GA
requested review from
robbibt,
BexDunn,
erialC-P,
uchchwhash,
Kooie-cate,
geoscience-aman,
margaretharrison,
vnewey,
Ariana-B,
amanda2099,
supermarkion and
erin-telfer
as code owners
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
robbibt
approved these changes
Jul 15, 2024
robbibt
added a commit
that referenced
this pull request
Feb 7, 2025
* Corrected 07_Parallel_processing_with_Dask to 09_Parallel_processing_with_Dask * Fixed the sandbox URL to go directly to the sandbox * Fixed 01_Jupyter_notebooks.ipynb link * Fixing links in Jupyter notebooks * Fixing links in Jupyter notebooks * Fixing links by converting them to relative links in Jupyter notebooks * Changed some URLs to absolute URLs to fix links * Changed notebook URLs to absolute URLs * Fixing links * Fixing links * Fixing a link * Changed data product links to prevent 403 redirects * Changed more product links to avoid 403 redirects * Removed anchors from links to data products * Fixing links * Fixing links * Fixed more links * Fixed many notebooks links * Fixing links * Fixing links * Fixed more URLs * Simplify tests * Remove indexing script, rename, remove DEA Tools install * Run on test branch * Restore DEA Tools install * Add remainder of DEA products and How to notebooks, increase durations * Add real world examples * [ci] Longer duration AWS creds for tests * [ci] Bump AWS ECR Login Action to now mask creds Now use v2, which masks creds by default. See aws-actions/amazon-ecr-login#526 * Feature/update explorer links (#1154) * Audit and update URL hyperlinks in Beginners Guide, DEA Products, How to Guides, and Interactive Maps notesbooks * Audit and update URL hyperlinks Real World Examples notesbooks * Rerun STAC notebook --------- Co-authored-by: robbibt <[email protected]> * Extend *both* AWS Role durations * Fixing broken links and images * Test real world examples only * Fix bugs in ML notebooks * Fixing spelling errors (cherry picked from commit d93d9e5) * Update coastal.py (#1155) * Added the sensor name Oli Tirs to the end of the Landsat 9 product URLs to make it consistent * Changing case of 'Tools' in links to fix broken links * Fixing some miscellaneous broken links * Fixing one link (casing) * Fixing 3 links * Fixed a waterbodies link * Reverted data_tools source (.py) links back to relative links so that they will open in a tab in the Jupyter Notebook * Reverted another data tools Python link * Reverted some remaining links to be .py links * Fixed one Scalable machine learning relative link * Fixed several more links - 19 Dec * Fixed notebooks links that were broken due to my previous changes (#1162) * Added the sensor name Oli Tirs to the end of the Landsat 9 product URLs to make it consistent * Changing case of 'Tools' in links to fix broken links * Fixing some miscellaneous broken links * Fixing one link (casing) * Fixing 3 links * Fixed a waterbodies link * Reverted data_tools source (.py) links back to relative links so that they will open in a tab in the Jupyter Notebook * Reverted another data tools Python link * Reverted some remaining links to be .py links * Fixed one Scalable machine learning relative link * Fixed several more links - 19 Dec --------- Co-authored-by: Benjamin Glitsos <[email protected]> * Fixed one link * Fixing one notebooks link * Fixed two anchor links * Deleted the root toctree since this is handled in the dea-docs git repo now * Removed all toctree directives because it is handled by the dea-docs toctree now, and replaced them with the sphinx-external-toc tableofcontents directive * Revert "Deleted the root toctree since this is handled in the dea-docs git repo now" This reverts commit 307d449. * Revert "Removed all toctree directives because it is handled by the dea-docs toctree now, and replaced them with the sphinx-external-toc tableofcontents directive" This reverts commit 04fb61c. * Edited the toctrees to align with the new Notebooks sidebar on the site * Edited the toctrees to align with the new Notebooks sidebar on the site (cherry picked from commit 01f5718) * Merged benji branch * Reverted to using a relative link for the DEA_notebooks_template because it is more important that this displays in a new tab properly in the Sandbox. Later I will try to fix this broken link on the dea-docs site * Add threads flag to parallel apply (#1171) * Add use_threads argument to parallel_apply * Cleanup imports * Update last modified date * Add brief explanation of threads vs processes * Add course to USAGE.rst (#1172) * Update USAGE.rst Add UNE course to courses list * Update spellcheck_wordlist.txt add 'automodule' to spellcheck word list and see if that helps * Remove spellcheck action and replace with PR checklist item (#1174) * Remove spellcheck action and replace with PR checklist item * Upload image directly * Delete .github/sandbox_spellchecker.jpg * Update README.rst (#1168) Partially addresses #1167 - add dea docs links - partially make the dev stable bit a bit clearer - update references to DEA Knowledge Hub from DEA User Guide * add Malan citation 25-01-2024 (#1176) Add citation to usage.rst * Limit tide data downloaded in tests (#1177) * Rearranging the README to have the wiki links at the top * Simplifying the content at the start of the README * Wrote and replaced section: Contributing to DEA Notebooks * Editing the intro section * Several edits to the README * Several edits to the README * Simplifying the intro text further * Editing README further * Added absolute links back to README because it needs to work on the KH * Fixed a strange number appearing in the heading of this article on the KH * Fixing some bullet points displaying incorrectly on the KH * Fixed two links to python Tools files * Add user-friendly landing page to DEA Sandbox (#1187) * Added Robbi's prototype landing page to this branch * Edited the intro text * Editing the landing page * Editing the Home prototype * Editing this template further * Minor punctuation * Update notebook and add images * Minor updates * Minor wording and formatting * Editing wording * Removed the DEA Notebooks KH tile because it is redundant * More wording changes * More wording changes * More wording changes * Added CSS class to the image grid tables to allow styling on the KH * Removed CSS classes because no longer needed * Minor updates prior to publication * Fix Tools link * Minor edits to wording * Added a 'Using the sidebar' image tile --------- Co-authored-by: Benjamin Glitsos <[email protected]> Co-authored-by: Benjamin Glitsos <[email protected]> * Update several notebooks to use cleaner `odc-geo` code (#1186) * Update several notebooks to use cleaner odc-geo approaches * Apply Black formatting * Update comments and rerun * Add missing dependency (#1190) * Added pip install link and instructions to the intro and reworded intro * Minor rewording * Migrated Git workflow out of README and fixed one link * Converted README format from .rst to .md * Added contribute anchor link * Added more spacing * Adjusting spacing * Formatting using em-dashes * Minor edits * Editing the README * Reworded introduction intro * Editing Introduction section * Updating to use KH links * Editing README further * Wrote separate section for the sandbox * Added DEA Sandbox image * Styling the image * Moved the image * Grammarly check * Review of draft * Updated directory descriptions to match the Sandbox's new cover page * Moved the introduction paragraph lower into its own section * Added NCI content * Added NCI content * Removed accidental link * Updating NCI and Sandbox links * Commented out the tags paragraph * Moving the Knowledge Hub section * Moving KH section back up * Minor wording * Minor formatting * Changed one link * Using absolute github links for readme images to fix broken images on KH until I create a specific homepage for KH Notebooks * Update README.md Co-authored-by: Robbi Bishop-Taylor <[email protected]> * Update README.md Co-authored-by: Robbi Bishop-Taylor <[email protected]> * Update USAGE.rst, Adding Citation (Issue #1178) (#1181) * Update USAGE.rst, Adding Citation (Issue #1178) * Update USAGE.rst --------- Co-authored-by: Bex Dunn <[email protected]> * Updated all Knowledge Hub 'docs' links to new 'knowledge' links * Updating all Knowledge Hub links in entire repo * Sphinx autosummary generated an update to the Tools * Renamed all 'Github' to 'GitHub' * Add updated xr_interpolate func and tests (#1205) * Add updated xr_interpolate func and tests * Rename test * Remove commented line * Update dea_tools.spatial.rst * Updated references to the old repo name 'dea-docs' to the new name (#1204) * Updated references to the old repo name 'dea-docs' to the new name * Updated the Known Issues link --------- Co-authored-by: Benjamin Glitsos <[email protected]> * Updates to DEA Waterbodies notebooks and tools for v3.0 (#1199) * start editing, change modified date * update v2 to v3 * update examples to v3 * add new image to DEA Waterbodies notebooks * rerun prior to PR * add @whatnick 's version support for v2 and v3 * change heading size for disclaimer text * include back compatibility in waterbodies.py script and pt and propagate to waterbodies notebooks * add header image and move disclaimer to info box * update links to knowledge hub * check knowledgehub links' * rollback to develop to fix merge conflict * reapply changes, and update knowledge hub links and GitHub references * having to roll back again because of merge conflicts * DEA Waterbodies notebook for v3, closes issue #1200 (#1207) * add citation to usage.rst (#1208) * add citation closes #1179 * Update USAGE.rst Co-authored-by: Robbi Bishop-Taylor <[email protected]> --------- Co-authored-by: Robbi Bishop-Taylor <[email protected]> * Merge develop to stable to add recent changes to support DEA Waterbodies v3.0 (#1209) * Updates to DEA Waterbodies notebooks and tools for v3.0 (#1199) * start editing, change modified date * update v2 to v3 * update examples to v3 * add new image to DEA Waterbodies notebooks * rerun prior to PR * add @whatnick 's version support for v2 and v3 * change heading size for disclaimer text * include back compatibility in waterbodies.py script and pt and propagate to waterbodies notebooks * add header image and move disclaimer to info box * update links to knowledge hub * check knowledgehub links' * rollback to develop to fix merge conflict * reapply changes, and update knowledge hub links and GitHub references * having to roll back again because of merge conflicts * DEA Waterbodies notebook for v3, closes issue #1200 (#1207) * add citation to usage.rst (#1208) * add citation closes #1179 * Update USAGE.rst Co-authored-by: Robbi Bishop-Taylor <[email protected]> --------- Co-authored-by: Robbi Bishop-Taylor <[email protected]> --------- Co-authored-by: Robbi Bishop-Taylor <[email protected]> * Change `xr_interpolate` extent Error to Warning (#1210) * Change `xr_interpolate` extent Error to Warning * Update tests * Update docstring * Update R2 method in validation code (#1211) * Introduction to DEA Intertidal product notebook (#1214) * Add files via upload Commits a new notebook introducing the DEA Intertidal product suite to the DEA_products sub-repository of dea-notebooks TODO - needs renaming to match the existing file naming convention * updated test database to include DEA_Intertidal dataset * PR ready notebook code formatted to 'Black' style, spelling checked, tags updates, outputs cleared and run from beginning, tests database updated to add new dataset * Commit the renamed notebook * Notebook updates * Use absolute dates * Index DEA Intertidal data into tests --------- Co-authored-by: robbibt <[email protected]> * Test new alert box format (#1216) * Update alert box format * Point DEA Intertidal KH link to product page, not notebook (#1219) * Removed 'sandbox' from the DEA Explorer URLs * add csv export to DEA WIT notebook (#1222) * add csv export to DEA WIT notebook * update link to sandbox doco to DEA Sandbox user guide * update masking link to directly point to WO bit flags * update links * updated links * fix dea explorer links * fix links and rerun * update Explorer links to remove sandbox from link text * remove ?/tab=overview from links: * reinstate relative link to Water Obs notebook * fixed the links I think thanks all * add link to function docs for dea-datahandling * rerun notebook in order * rerun notebook in order and check links * update last run date * Adding 'products' segment to the DEA Explorer URLs * Update index.rst (#1227) add wetlands to index.rst * Deleted the 'gen' folder which is auto-generated at runtime by Sphinx autosummary * Moved the mock_imports file into the dea-notebooks repo * Moved the file up one level of the folder tree * Moved the file back to where it was before * Reworded code comment for accuracy * Add updated `xr_regression` function for multi-dimensional linear regression (#1226) * Add updated xr_regression function * Add dask support for lazy computation * Set dtypes * Update docstring * Update docstring * Add MAD outliers * Update docstring * Remove lag functionality * Update docstrings * Add better error handling * Update stream gauge corr notebook to use new func * Adding DEAfrica Wetland Turbidity notebook for Australian study site (#1175) * Adding DEAfrica Wetland Turbidity notebook for Australian study site * change all instances of NDTI to NDTI2 to reflect usage at top of notebook * update notebook to use Collection 3 WO Statistics * rerun notebook --------- Co-authored-by: BexDunn <[email protected]> * Add spatial interpolation with `xr_interpolate` notebook (#1233) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook * Remove coastal files from branch * Add points data * Review feedback; * Add p param to IDW * Fix test * update GeoMAD with latest version * Updates to product notebook Knowledge Hub links and DEA notebook content (#1221) * Move KH links into consistent alert box format * Update DEA notebook * Minor wording updates * Minor wording * Temporarily remove STAC notebook from tests * Add ensemble tide modelling functionality to `model_tides` (#1231) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook * Publish recent changes to `stable` branch (#1234) * Add updated `xr_regression` function for multi-dimensional linear regression (#1226) * Add updated xr_regression function * Add dask support for lazy computation * Set dtypes * Update docstring * Update docstring * Add MAD outliers * Update docstring * Remove lag functionality * Update docstrings * Add better error handling * Update stream gauge corr notebook to use new func * Adding DEAfrica Wetland Turbidity notebook for Australian study site (#1175) * Adding DEAfrica Wetland Turbidity notebook for Australian study site * change all instances of NDTI to NDTI2 to reflect usage at top of notebook * update notebook to use Collection 3 WO Statistics * rerun notebook --------- Co-authored-by: BexDunn <[email protected]> * Add spatial interpolation with `xr_interpolate` notebook (#1233) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook * Remove coastal files from branch * Add points data * Review feedback; * Add p param to IDW * Fix test * Updates to product notebook Knowledge Hub links and DEA notebook content (#1221) * Move KH links into consistent alert box format * Update DEA notebook * Minor wording updates * Minor wording * Temporarily remove STAC notebook from tests * Add ensemble tide modelling functionality to `model_tides` (#1231) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook --------- Co-authored-by: Matt-dea <[email protected]> Co-authored-by: BexDunn <[email protected]> * update tests to use new GeoMAD products * update export netcdf notebook with new GeoMAD * Update setup_test_datacube.sh * Update setup_test_datacube.sh Index based on STAC doc instead * Update setup_test_datacube.sh * Update to new c3 grid path rows * Update loading_data notebook to use new GeoMAD * update plotting notebook with new GeoMAD * Update parallel processing notebook with new GeoMAD * update Animated_timeseries with new GeoMAD * Update DEA_Intertidal with new GeoMAD * Update DEA_GeoMAD.ipynb * Expand GeoMAD test tiles * Update 1_Extract_training_data.ipynb * Update 04_Loading_data.ipynb * Update 09_Parallel_processing_with_Dask.ipynb * Update DEA_Intertidal.ipynb * Update Exporting_NetCDFs.ipynb * Update Animated_timeseries.ipynb * Update 05_Plotting.ipynb * Update 09_Parallel_processing_with_Dask.ipynb * Temporarily modify spatial.py tests to work around odc-geo bug (#1235) * Fix bug affecting latest Sandbox image, update notebook code (#1237) * Fix bug in intertidal notebook, update * Fix missing test data * Fix WIT bug * Fix seasonality notebook * reorganize dask compute to avoid repeated load in wetland turbidity notebook (#1240) Co-authored-by: uchchwhash <[email protected]> * Update change detection notebook (#1241) * Update change detection notebook * Fix notebook * Fix notebook * Update notebook (#1242) * Notebook runtime improvements (#1245) * Update Seasonal_water_extents.ipynb * Update Generating_geomedian_composites.ipynb * Update Land_cover_animated_plots.ipynb * Reduced training data polygons by half to improve run time by 50 percent (#1243) * Updating descriptions on the 'category pages' with new wording (#1246) * Updated introduction: Beginner's guide * Updated introduction: Real-world examples * Updated introduction: DEA Tools Package * Speed up WIT notebook (#1244) * add dask chunks * speedup and some edits * genericise dask chunks * Update tide modelling notebook to be faster, add support for FES2022 (#1247) * Add FES2022, update notebook to be faster * Fix study area * Add support for model cropping within tide modelling functions (#1249) * Add support for model cropping within tide modelling functions * Replace accidental file * WIT note about loading polygons * Improve accuracy of NOTE message * Added GeoMAD to the index file * Edited the title to conform to consistent title format * Rearranged order of notebooks in sidebar to match the Knowledge Hub products sidebar * Renamed Github trigger event * Moved GeoMAD notebook in the sidebar to reflect the order of products in (#1255) the KH * Minor updates for NCI environment compatibility (#1256) * Update test_temporal.py * Skip broken datasets * Skip broken datasets * Skip broken datasets * Fix docker compose in workflow * Missing docker compose * Update test_temporal.py * Updated all ODC Slack links to the new ODC Discord link (#1258) * Moved GeoMAD notebook in the sidebar to reflect the order of products in the KH * Three discord links * 3 more Discord links * Minor case formatting change * Multiple Discord links * More Discord links * Updated numerous links to discord using find-and-replace * Two Discord links * Multiple Discord links - find-and-replace * Multiple discord links * Add Discord badge * Use consistent link format --------- Co-authored-by: Robbi Bishop-Taylor <[email protected]> * Fix deprecation warnings and speed up code * Change last modified date * Intertidal exposure (#1261) * New files for intertidal exposure notebook * minor editing to Load packages cells * Updated discord and removed slack links * Improved markdown linking to image and gif in Introduction * Incorporated SS and MA reviews * Updated to include RBT reviews * Minor pip install and notebook naming edits. Add notebook to README * adding renamed exposure notebook back into PR * adding global SAR access through microsoft planetary compute (#1263) * adding global SAR access through microsoft planetary compute * Make minor spelling and formatting amendments. * small changes for PR --------- Co-authored-by: geoscience-aman <[email protected]> * Update USAGE.rst (#1268) Add Swinburne course for 2024 * Minor compatibility change for tide modelling package (#1269) * Mitch predict_xr (#1270) * add probability array output to predict_xr * predict_xr at proba_max args * predict_xr match arg names * xr_predict deal with multiband prob outout * xr_predict merge output probs * clean up comments and spacing * Update USAGE.rst (#1272) Add new reference, Burton et al 2024 Enhancing long-term vegetation monitoring in Australia: a new approach for harmonising the Advanced Very High Resolution Radiometer normalised-difference vegetation (NVDI) with MODIS NDVI * Fix broken code on `unstable` Sandbox image (#1274) * Updates for pyTMD * Fix contours bug due to groupby squeeze * Try loosening pyTMD requirements * Update tests to pass on both stable and unstable sandbox * Fix pansharpening bug * Deprecate tide modelling tools in favour of `eo-tides` (#1276) * small fixes in input for calculate_indices() rgb() added line to prevent warnings from dask. Changed collection name in calculate_indices(), and reset band names in rgb() to default to avoid errors * test * revert last test commit * Revert "revert last test commit" revert last test commit This reverts commit 7d8d503. * revert test commit This reverts commit adddf58. * product and collection names fixed * Update product and band names in ML notebook `Machine_learning_with_ODC` (#1276) (#1278) * update product and band names used in notebook * update geoMAD product and band names * change hyperlink to match updated geoMAD product * Fix data sync --------- Co-authored-by: Benjamin Glitsos <[email protected]> Co-authored-by: Damien Ayers <[email protected]> Co-authored-by: Matthew Fry <[email protected]> Co-authored-by: ClaireP <[email protected]> Co-authored-by: Benjamin Glitsos <[email protected]> Co-authored-by: Alex Leith <[email protected]> Co-authored-by: Bex Dunn <[email protected]> Co-authored-by: Benjamin Glitsos <[email protected]> Co-authored-by: Nicholas Cimino <[email protected]> Co-authored-by: Matt-dea <[email protected]> Co-authored-by: BexDunn <[email protected]> Co-authored-by: James Miller <[email protected]> Co-authored-by: James Miller <[email protected]> Co-authored-by: Imam Alam <[email protected]> Co-authored-by: uchchwhash <[email protected]> Co-authored-by: LaurenSchenk1 <[email protected]> Co-authored-by: LaurenSchenk1 <[email protected]> Co-authored-by: Aman Chopra <[email protected]> Co-authored-by: geoscience-aman <[email protected]> Co-authored-by: Alex Bradley <[email protected]> Co-authored-by: Mitchell Lyons <[email protected]> Co-authored-by: Gianluca S. <[email protected]> Co-authored-by: Gianluca Scortechini <[email protected]> Co-authored-by: Jenna Guffogg <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Update AOI to reduce notebook runtimes