Skip to content

Commit

Permalink
Publish changes to develop to DEA Sandbox and Knowledge Hub (#1281)
Browse files Browse the repository at this point in the history
* Fix deprecation warnings and speed up code

* Change last modified date

* Intertidal exposure (#1261)

* New files for intertidal exposure notebook

* minor editing to Load packages cells

* Updated discord and removed slack links

* Improved markdown linking to image and gif in Introduction

* Incorporated SS and MA reviews

* Updated to include RBT reviews

* Minor pip install and notebook naming edits. Add notebook to README

* adding renamed exposure notebook back into PR

* adding global SAR access through microsoft planetary compute (#1263)

* adding global SAR access through microsoft planetary compute

* Make minor spelling and formatting amendments.

* small changes for PR

---------

Co-authored-by: geoscience-aman <[email protected]>

* Update USAGE.rst (#1268)

Add Swinburne course for 2024

* Minor compatibility change for tide modelling package (#1269)

* Mitch predict_xr (#1270)

* add probability array output to predict_xr

* predict_xr at proba_max args

* predict_xr match arg names

* xr_predict deal with multiband prob outout

* xr_predict merge output probs

* clean up comments and spacing

* Update USAGE.rst (#1272)

Add new reference, Burton et al 2024 Enhancing long-term vegetation monitoring in Australia: a new approach for harmonising the Advanced Very High Resolution Radiometer normalised-difference vegetation (NVDI) with MODIS NDVI

* Fix broken code on `unstable` Sandbox image (#1274)

* Updates for pyTMD

* Fix contours bug due to groupby squeeze

* Try loosening pyTMD requirements

* Update tests to pass on both stable and unstable sandbox

* Fix pansharpening bug

* Deprecate tide modelling tools in favour of `eo-tides` (#1276)

* small fixes in input for calculate_indices() rgb()

added line to prevent warnings from dask.
Changed collection name in calculate_indices(), and reset band names in rgb() to default to avoid errors

* test

* revert last test commit

* Revert "revert last test commit"
revert last test commit
This reverts commit 7d8d503.

* revert test commit

This reverts commit adddf58.

* product and collection names fixed

* Update product and band names in ML notebook `Machine_learning_with_ODC` (#1276) (#1278)

* update product and band names used in notebook

* update geoMAD product and band names

* change hyperlink to match updated geoMAD product

* Implement comprehensive GitHub Actions integration tests using prod DEA ODC database (#1152)

* [gha] Test Remote Datacube Connection

* Fix Micromamba setup

* Test S3 Access and RDS Access

* attempt to run the whole kit and kaboodle

* Fix GHA Action v numbers

* Whoops, missed a slash

* Magic to access the port outside the docker container

* Run on host network to access RDS Port Forward

* Updates to database access PR (#1153)

* Corrected 07_Parallel_processing_with_Dask to 09_Parallel_processing_with_Dask

* Fixed the sandbox URL to go directly to the sandbox

* Fixed 01_Jupyter_notebooks.ipynb link

* Fixing links in Jupyter notebooks

* Fixing links in Jupyter notebooks

* Fixing links by converting them to relative links in Jupyter notebooks

* Changed some URLs to absolute URLs to fix links

* Changed notebook URLs to absolute URLs

* Fixing links

* Fixing links

* Fixing a link

* Changed data product links to prevent 403 redirects

* Changed more product links to avoid 403 redirects

* Removed anchors from links to data products

* Fixing links

* Fixing links

* Fixed more links

* Fixed many notebooks links

* Fixing links

* Fixing links

* Fixed more URLs

* Simplify tests

* Remove indexing script, rename, remove DEA Tools install

* Run on test branch

* Restore DEA Tools install

* Add remainder of DEA products and How to notebooks, increase durations

* Add real world examples

* [ci] Longer duration AWS creds for tests

* [ci] Bump AWS ECR Login Action to now mask creds

Now use v2, which masks creds by default.
See aws-actions/amazon-ecr-login#526

* Feature/update explorer links (#1154)

* Audit and update URL hyperlinks in Beginners Guide, DEA Products, How to Guides, and Interactive Maps notesbooks

* Audit and update URL hyperlinks Real World Examples notesbooks

* Rerun STAC notebook

---------

Co-authored-by: robbibt <[email protected]>

* Extend *both* AWS Role durations

* Fixing broken links and images

* Test real world examples only

* Fix bugs in ML notebooks

* Fixing spelling errors

(cherry picked from commit d93d9e5)

* Update coastal.py (#1155)

* Added the sensor name Oli Tirs to the end of the Landsat 9 product URLs to make it consistent

* Changing case of 'Tools' in links to fix broken links

* Fixing some miscellaneous broken links

* Fixing one link (casing)

* Fixing 3 links

* Fixed a waterbodies link

* Reverted data_tools source (.py) links back to relative links so that they will open in a tab in the Jupyter Notebook

* Reverted another data tools Python link

* Reverted some remaining links to be .py links

* Fixed one Scalable machine learning relative link

* Fixed several more links - 19 Dec

* Fixed notebooks links that were broken due to my previous changes (#1162)

* Added the sensor name Oli Tirs to the end of the Landsat 9 product URLs to make it consistent

* Changing case of 'Tools' in links to fix broken links

* Fixing some miscellaneous broken links

* Fixing one link (casing)

* Fixing 3 links

* Fixed a waterbodies link

* Reverted data_tools source (.py) links back to relative links so that they will open in a tab in the Jupyter Notebook

* Reverted another data tools Python link

* Reverted some remaining links to be .py links

* Fixed one Scalable machine learning relative link

* Fixed several more links - 19 Dec

---------

Co-authored-by: Benjamin Glitsos <[email protected]>

* Fixed one link

* Fixing one notebooks link

* Fixed two anchor links

* Deleted the root toctree since this is handled in the dea-docs git repo now

* Removed all toctree directives because it is handled by the dea-docs toctree now, and replaced them with the sphinx-external-toc tableofcontents directive

* Revert "Deleted the root toctree since this is handled in the dea-docs git repo now"

This reverts commit 307d449.

* Revert "Removed all toctree directives because it is handled by the dea-docs toctree now, and replaced them with the sphinx-external-toc tableofcontents directive"

This reverts commit 04fb61c.

* Edited the toctrees to align with the new Notebooks sidebar on the site

* Edited the toctrees to align with the new Notebooks sidebar on the site

(cherry picked from commit 01f5718)

* Merged benji branch

* Reverted to using a relative link for the DEA_notebooks_template because it is more important that this displays in a new tab properly in the Sandbox. Later I will try to fix this broken link on the dea-docs site

* Add threads flag to parallel apply (#1171)

* Add use_threads argument to parallel_apply

* Cleanup imports

* Update last modified date

* Add brief explanation of threads vs processes

* Add course to USAGE.rst (#1172)

* Update USAGE.rst

Add UNE course to courses list

* Update spellcheck_wordlist.txt

add 'automodule' to spellcheck word list and see if that helps

* Remove spellcheck action and replace with PR checklist item (#1174)

* Remove spellcheck action and replace with PR checklist item

* Upload image directly

* Delete .github/sandbox_spellchecker.jpg

* Update README.rst (#1168)

Partially addresses #1167 
- add dea docs links
- partially make the dev stable bit a bit clearer
- update references to DEA Knowledge Hub from DEA User Guide

* add Malan citation 25-01-2024 (#1176)

Add citation to usage.rst

* Limit tide data downloaded in tests (#1177)

* Rearranging the README to have the wiki links at the top

* Simplifying the content at the start of the README

* Wrote and replaced section: Contributing to DEA Notebooks

* Editing the intro section

* Several edits to the README

* Several edits to the README

* Simplifying the intro text further

* Editing README further

* Added absolute links back to README because it needs to work on the KH

* Fixed a strange number appearing in the heading of this article on the KH

* Fixing some bullet points displaying incorrectly on the KH

* Fixed two links to python Tools files

* Add user-friendly landing page to DEA Sandbox (#1187)

* Added Robbi's prototype landing page to this branch

* Edited the intro text

* Editing the landing page

* Editing the Home prototype

* Editing this template further

* Minor punctuation

* Update notebook and add images

* Minor updates

* Minor wording and formatting

* Editing wording

* Removed the DEA Notebooks KH tile because it is redundant

* More wording changes

* More wording changes

* More wording changes

* Added CSS class to the image grid tables to allow styling on the KH

* Removed CSS classes because no longer needed

* Minor updates prior to publication

* Fix Tools link

* Minor edits to wording

* Added a 'Using the sidebar' image tile

---------

Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>

* Update several notebooks to use cleaner `odc-geo` code (#1186)

* Update several notebooks to use cleaner odc-geo approaches

* Apply Black formatting

* Update comments and rerun

* Add missing dependency (#1190)

* Added pip install link and instructions to the intro and reworded intro

* Minor rewording

* Migrated Git workflow out of README and fixed one link

* Converted README format from .rst to .md

* Added contribute anchor link

* Added more spacing

* Adjusting spacing

* Formatting using em-dashes

* Minor edits

* Editing the README

* Reworded introduction intro

* Editing Introduction section

* Updating to use KH links

* Editing README further

* Wrote separate section for the sandbox

* Added DEA Sandbox image

* Styling the image

* Moved the image

* Grammarly check

* Review of draft

* Updated directory descriptions to match the Sandbox's new cover page

* Moved the introduction paragraph lower into its own section

* Added NCI content

* Added NCI content

* Removed accidental link

* Updating NCI and Sandbox links

* Commented out the tags paragraph

* Moving the Knowledge Hub section

* Moving KH section back up

* Minor wording

* Minor formatting

* Changed one link

* Using absolute github links for readme images to fix broken images on KH until I create a specific homepage for KH Notebooks

* Update README.md

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Update README.md

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Update USAGE.rst, Adding Citation (Issue #1178) (#1181)

* Update USAGE.rst, Adding Citation (Issue #1178)

* Update USAGE.rst

---------

Co-authored-by: Bex Dunn <[email protected]>

* Updated all Knowledge Hub 'docs' links to new 'knowledge' links

* Updating all Knowledge Hub links in entire repo

* Sphinx autosummary generated an update to the Tools

* Renamed all 'Github' to 'GitHub'

* Add updated xr_interpolate func and tests (#1205)

* Add updated xr_interpolate func and tests

* Rename test

* Remove commented line

* Update dea_tools.spatial.rst

* Updated references to the old repo name 'dea-docs' to the new name (#1204)

* Updated references to the old repo name 'dea-docs' to the new name

* Updated the Known Issues link

---------

Co-authored-by: Benjamin Glitsos <[email protected]>

* Updates to DEA Waterbodies notebooks and tools for v3.0 (#1199)

* start editing, change modified date

* update v2 to v3

* update examples to v3

* add new image to DEA Waterbodies notebooks

* rerun prior to PR

* add @whatnick 's version support for v2 and v3

* change heading size for disclaimer text

* include back compatibility in waterbodies.py script and pt and propagate to waterbodies notebooks

* add header image and move disclaimer to info box

* update links to knowledge hub

* check knowledgehub links'

* rollback to develop to fix merge conflict

* reapply changes, and update knowledge hub links and GitHub references

* having to roll back again because of merge conflicts

* DEA Waterbodies notebook for v3, closes issue #1200 (#1207)

* add citation to usage.rst (#1208)

* add citation 

closes #1179

* Update USAGE.rst

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

---------

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Merge develop to stable to add recent changes to support DEA Waterbodies v3.0 (#1209)

* Updates to DEA Waterbodies notebooks and tools for v3.0 (#1199)

* start editing, change modified date

* update v2 to v3

* update examples to v3

* add new image to DEA Waterbodies notebooks

* rerun prior to PR

* add @whatnick 's version support for v2 and v3

* change heading size for disclaimer text

* include back compatibility in waterbodies.py script and pt and propagate to waterbodies notebooks

* add header image and move disclaimer to info box

* update links to knowledge hub

* check knowledgehub links'

* rollback to develop to fix merge conflict

* reapply changes, and update knowledge hub links and GitHub references

* having to roll back again because of merge conflicts

* DEA Waterbodies notebook for v3, closes issue #1200 (#1207)

* add citation to usage.rst (#1208)

* add citation 

closes #1179

* Update USAGE.rst

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

---------

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

---------

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Change `xr_interpolate` extent Error to Warning (#1210)

* Change `xr_interpolate` extent Error to Warning

* Update tests

* Update docstring

* Update R2 method in validation code (#1211)

* Introduction to DEA Intertidal product notebook (#1214)

* Add files via upload

Commits a new notebook introducing the DEA Intertidal product suite to the DEA_products sub-repository of dea-notebooks
TODO - needs renaming to match the existing file naming convention

* updated test database to include DEA_Intertidal dataset

* PR ready notebook

code formatted to 'Black' style, spelling checked, tags updates, outputs cleared and run from beginning, tests database updated to add new dataset

* Commit the renamed notebook

* Notebook updates

* Use absolute dates

* Index DEA Intertidal data into tests

---------

Co-authored-by: robbibt <[email protected]>

* Test new alert box format (#1216)

* Update alert box format

* Point DEA Intertidal KH link to product page, not notebook (#1219)

* Removed 'sandbox' from the DEA Explorer URLs

* add csv export to DEA WIT notebook (#1222)

* add csv export to DEA WIT notebook

* update link to sandbox doco to DEA Sandbox user guide

* update masking link to directly point to WO bit flags

* update links

* updated links

* fix dea explorer links

* fix links and rerun

* update Explorer links to remove sandbox from link text

* remove ?/tab=overview from links:

* reinstate relative link to Water Obs notebook

* fixed the links I think thanks all

* add link to function docs for dea-datahandling

* rerun notebook in order

* rerun notebook in order and check links

* update last run date

* Adding 'products' segment to the DEA Explorer URLs

* Update index.rst (#1227)

add wetlands to index.rst

* Deleted the 'gen' folder which is auto-generated at runtime by Sphinx autosummary

* Moved the mock_imports file into the dea-notebooks repo

* Moved the file up one level of the folder tree

* Moved the file back to where it was before

* Reworded code comment for accuracy

* Add updated `xr_regression` function for multi-dimensional linear regression (#1226)

* Add updated xr_regression function

* Add dask support for lazy computation

* Set dtypes

* Update docstring

* Update docstring

* Add MAD outliers

* Update docstring

* Remove lag functionality

* Update docstrings

* Add better error handling

* Update stream gauge corr notebook to use new func

* Adding DEAfrica Wetland Turbidity notebook for Australian study site (#1175)

* Adding DEAfrica Wetland Turbidity notebook for Australian study site

* change all instances of NDTI to NDTI2 to reflect usage at top of notebook

* update notebook to use Collection 3 WO Statistics

* rerun notebook

---------

Co-authored-by: BexDunn <[email protected]>

* Add spatial interpolation with `xr_interpolate` notebook (#1233)

* Add ensemble tide modelling functionality to model_tides

* Update test_coastal.py

* Remove test

* Updates to IDW, xr_interpolate and ensemble tide modelling code"

* Doco updates

* Switch ensemble rankings from high to low = good

* Update docstring

* Fix doco

* Add interpolation notebook

* Remove coastal files from branch

* Add points data

* Review feedback;

* Add p param to IDW

* Fix test

* update GeoMAD with latest version

* Updates to product notebook Knowledge Hub links and DEA notebook content (#1221)

* Move KH links into consistent alert box format

* Update DEA notebook

* Minor wording updates

* Minor wording

* Temporarily remove STAC notebook from tests

* Add ensemble tide modelling functionality to `model_tides` (#1231)

* Add ensemble tide modelling functionality to model_tides

* Update test_coastal.py

* Remove test

* Updates to IDW, xr_interpolate and ensemble tide modelling code"

* Doco updates

* Switch ensemble rankings from high to low = good

* Update docstring

* Fix doco

* Add interpolation notebook

* Publish recent changes to `stable` branch (#1234)

* Add updated `xr_regression` function for multi-dimensional linear regression (#1226)

* Add updated xr_regression function

* Add dask support for lazy computation

* Set dtypes

* Update docstring

* Update docstring

* Add MAD outliers

* Update docstring

* Remove lag functionality

* Update docstrings

* Add better error handling

* Update stream gauge corr notebook to use new func

* Adding DEAfrica Wetland Turbidity notebook for Australian study site (#1175)

* Adding DEAfrica Wetland Turbidity notebook for Australian study site

* change all instances of NDTI to NDTI2 to reflect usage at top of notebook

* update notebook to use Collection 3 WO Statistics

* rerun notebook

---------

Co-authored-by: BexDunn <[email protected]>

* Add spatial interpolation with `xr_interpolate` notebook (#1233)

* Add ensemble tide modelling functionality to model_tides

* Update test_coastal.py

* Remove test

* Updates to IDW, xr_interpolate and ensemble tide modelling code"

* Doco updates

* Switch ensemble rankings from high to low = good

* Update docstring

* Fix doco

* Add interpolation notebook

* Remove coastal files from branch

* Add points data

* Review feedback;

* Add p param to IDW

* Fix test

* Updates to product notebook Knowledge Hub links and DEA notebook content (#1221)

* Move KH links into consistent alert box format

* Update DEA notebook

* Minor wording updates

* Minor wording

* Temporarily remove STAC notebook from tests

* Add ensemble tide modelling functionality to `model_tides` (#1231)

* Add ensemble tide modelling functionality to model_tides

* Update test_coastal.py

* Remove test

* Updates to IDW, xr_interpolate and ensemble tide modelling code"

* Doco updates

* Switch ensemble rankings from high to low = good

* Update docstring

* Fix doco

* Add interpolation notebook

---------

Co-authored-by: Matt-dea <[email protected]>
Co-authored-by: BexDunn <[email protected]>

* update tests to use new GeoMAD products

* update export netcdf notebook with new GeoMAD

* Update setup_test_datacube.sh

* Update setup_test_datacube.sh

Index based on STAC doc instead

* Update setup_test_datacube.sh

* Update to new c3 grid path rows

* Update loading_data notebook to use new GeoMAD

* update plotting notebook with new GeoMAD

* Update parallel processing notebook with new GeoMAD

* update Animated_timeseries with new GeoMAD

* Update DEA_Intertidal with new GeoMAD

* Update DEA_GeoMAD.ipynb

* Expand GeoMAD test tiles

* Update 1_Extract_training_data.ipynb

* Update 04_Loading_data.ipynb

* Update 09_Parallel_processing_with_Dask.ipynb

* Update DEA_Intertidal.ipynb

* Update Exporting_NetCDFs.ipynb

* Update Animated_timeseries.ipynb

* Update 05_Plotting.ipynb

* Update 09_Parallel_processing_with_Dask.ipynb

* Temporarily modify spatial.py tests to work around odc-geo bug (#1235)

* Fix bug affecting latest Sandbox image, update notebook code (#1237)

* Fix bug in intertidal notebook, update

* Fix missing test data

* Fix WIT bug

* Fix seasonality notebook

* reorganize dask compute to avoid repeated load in wetland turbidity notebook (#1240)

Co-authored-by: uchchwhash <[email protected]>

* Update change detection notebook (#1241)

* Update change detection notebook

* Fix notebook

* Fix notebook

* Update notebook (#1242)

* Notebook runtime improvements (#1245)

* Update Seasonal_water_extents.ipynb

* Update Generating_geomedian_composites.ipynb

* Update Land_cover_animated_plots.ipynb

* Reduced training data polygons by half to improve run time by 50 percent (#1243)

* Updating descriptions on the 'category pages' with new wording (#1246)

* Updated introduction: Beginner's guide

* Updated introduction: Real-world examples

* Updated introduction: DEA Tools Package

* Speed up WIT notebook (#1244)

* add dask chunks

* speedup and some edits

* genericise dask chunks

* Update tide modelling notebook to be faster, add support for FES2022 (#1247)

* Add FES2022, update notebook to be faster

* Fix study area

* Add support for model cropping within tide modelling functions (#1249)

* Add support for model cropping within tide modelling functions

* Replace accidental file

* WIT note about loading polygons

* Improve accuracy of NOTE message

* Added GeoMAD to the index file

* Edited the title to conform to consistent title format

* Rearranged order of notebooks in sidebar to match the Knowledge Hub
products sidebar

* Renamed Github trigger event

* Moved GeoMAD notebook in the sidebar to reflect the order of products in (#1255)

the KH

* Minor updates for NCI environment compatibility (#1256)

* Update test_temporal.py

* Skip broken datasets

* Skip broken datasets

* Skip broken datasets

* Fix docker compose in workflow

* Missing docker compose

* Update test_temporal.py

* Updated all ODC Slack links to the new ODC Discord link (#1258)

* Moved GeoMAD notebook in the sidebar to reflect the order of products in
the KH

* Three discord links

* 3 more Discord links

* Minor case formatting change

* Multiple Discord links

* More Discord links

* Updated numerous links to discord using find-and-replace

* Two Discord links

* Multiple Discord links - find-and-replace

* Multiple discord links

* Add Discord badge

* Use consistent link format

---------

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Fix deprecation warnings and speed up code

* Change last modified date

* Intertidal exposure (#1261)

* New files for intertidal exposure notebook

* minor editing to Load packages cells

* Updated discord and removed slack links

* Improved markdown linking to image and gif in Introduction

* Incorporated SS and MA reviews

* Updated to include RBT reviews

* Minor pip install and notebook naming edits. Add notebook to README

* adding renamed exposure notebook back into PR

* adding global SAR access through microsoft planetary compute (#1263)

* adding global SAR access through microsoft planetary compute

* Make minor spelling and formatting amendments.

* small changes for PR

---------

Co-authored-by: geoscience-aman <[email protected]>

* Update USAGE.rst (#1268)

Add Swinburne course for 2024

* Minor compatibility change for tide modelling package (#1269)

* Mitch predict_xr (#1270)

* add probability array output to predict_xr

* predict_xr at proba_max args

* predict_xr match arg names

* xr_predict deal with multiband prob outout

* xr_predict merge output probs

* clean up comments and spacing

* Update USAGE.rst (#1272)

Add new reference, Burton et al 2024 Enhancing long-term vegetation monitoring in Australia: a new approach for harmonising the Advanced Very High Resolution Radiometer normalised-difference vegetation (NVDI) with MODIS NDVI

* Fix broken code on `unstable` Sandbox image (#1274)

* Updates for pyTMD

* Fix contours bug due to groupby squeeze

* Try loosening pyTMD requirements

* Update tests to pass on both stable and unstable sandbox

* Fix pansharpening bug

* Deprecate tide modelling tools in favour of `eo-tides` (#1276)

* small fixes in input for calculate_indices() rgb()

added line to prevent warnings from dask.
Changed collection name in calculate_indices(), and reset band names in rgb() to default to avoid errors

* test

* revert last test commit

* Revert "revert last test commit"
revert last test commit
This reverts commit 7d8d503.

* revert test commit

This reverts commit adddf58.

* product and collection names fixed

* Update product and band names in ML notebook `Machine_learning_with_ODC` (#1276) (#1278)

* update product and band names used in notebook

* update geoMAD product and band names

* change hyperlink to match updated geoMAD product

* Fix data sync

---------

Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Damien Ayers <[email protected]>
Co-authored-by: Matthew Fry <[email protected]>
Co-authored-by: ClaireP <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Alex Leith <[email protected]>
Co-authored-by: Bex Dunn <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Nicholas Cimino <[email protected]>
Co-authored-by: Matt-dea <[email protected]>
Co-authored-by: BexDunn <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: Imam Alam <[email protected]>
Co-authored-by: uchchwhash <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
Co-authored-by: Aman Chopra <[email protected]>
Co-authored-by: geoscience-aman <[email protected]>
Co-authored-by: Alex Bradley <[email protected]>
Co-authored-by: Mitchell Lyons <[email protected]>
Co-authored-by: Gianluca S. <[email protected]>
Co-authored-by: Gianluca Scortechini <[email protected]>
Co-authored-by: Jenna Guffogg <[email protected]>

* Revert notebook edits

* Test cron

* Move content to action

* Fix action

* Fix path

* Fix shell

* Try matrix

* Relocate files

* Add cron to action

* Add additional tests

* Fix names

* Test custom params

* Test passing empty value

* Add RWE

* Cleanup

* Formatting

* Wording

* Update test_notebooks.sh

* Expand RWE

* Update tests to add modified check

* Only run tests if notebooks modified

* Temporarily add PR trigger

* Pass in custom paths

* Wording

* Formatting update

* Fix formatting

* Formatting

* Update broken notebooks

* Fix other notebook issues

* Add in DEA Tools test

* Fix tests

* Separate DEA Tools and notebook tests

* Reduce run time for training point notebook

* Try without Docker compose

---------

Co-authored-by: Robbi Bishop-Taylor <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Matthew Fry <[email protected]>
Co-authored-by: ClaireP <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Alex Leith <[email protected]>
Co-authored-by: Bex Dunn <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Nicholas Cimino <[email protected]>
Co-authored-by: Matt-dea <[email protected]>
Co-authored-by: BexDunn <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: Imam Alam <[email protected]>
Co-authored-by: uchchwhash <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
Co-authored-by: Aman Chopra <[email protected]>
Co-authored-by: geoscience-aman <[email protected]>
Co-authored-by: Alex Bradley <[email protected]>
Co-authored-by: Mitchell Lyons <[email protected]>
Co-authored-by: Gianluca S. <[email protected]>
Co-authored-by: Gianluca Scortechini <[email protected]>
Co-authored-by: Jenna Guffogg <[email protected]>

* Try removing PR trigger for tests

* WIT Notebook fix csv output (#1280)

* fix csv output to include date column and remove index value

* simplify output code

* Rerun notebook, add last run date

* Update test schedule

* Update test action URL

* Update badge label

---------

Co-authored-by: robbibt <[email protected]>

---------

Co-authored-by: Aman Chopra <[email protected]>
Co-authored-by: geoscience-aman <[email protected]>
Co-authored-by: ClaireP <[email protected]>
Co-authored-by: Alex Bradley <[email protected]>
Co-authored-by: Bex Dunn <[email protected]>
Co-authored-by: Mitchell Lyons <[email protected]>
Co-authored-by: Gianluca S. <[email protected]>
Co-authored-by: Gianluca Scortechini <[email protected]>
Co-authored-by: Jenna Guffogg <[email protected]>
Co-authored-by: Damien Ayers <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Matthew Fry <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Alex Leith <[email protected]>
Co-authored-by: Benjamin Glitsos <[email protected]>
Co-authored-by: Nicholas Cimino <[email protected]>
Co-authored-by: Matt-dea <[email protected]>
Co-authored-by: BexDunn <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: James Miller <[email protected]>
Co-authored-by: Imam Alam <[email protected]>
Co-authored-by: uchchwhash <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
Co-authored-by: LaurenSchenk1 <[email protected]>
  • Loading branch information
25 people authored Feb 12, 2025
1 parent 8d22060 commit 192d4d5
Show file tree
Hide file tree
Showing 17 changed files with 1,558 additions and 1,142 deletions.
65 changes: 65 additions & 0 deletions .github/actions/test_setup/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: "Setup tests"
description: "Set up database access and tests"

inputs:
image:
description: "Sandbox image to test against"
required: true
default: "stable"

runs:
using: "composite"
steps:

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::538673716275:role/github-actions-role-readonly
aws-region: ap-southeast-2
role-duration-seconds: 7200 # 2 hours

- name: Copy tide modelling files with the AWS CLI
run: |
aws s3 sync s3://dea-non-public-data/tide_models/tide_models/fes2014 tide_models/fes2014
aws s3 sync s3://dea-non-public-data/tide_models/tide_models/hamtide tide_models/hamtide
shell: bash

- name: Login to Amazon ECR Private
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2

- name: Pull dea-sandbox image from ECR
run: |
docker pull 538673716275.dkr.ecr.ap-southeast-2.amazonaws.com/geoscienceaustralia/sandbox:${{ inputs.image }}
shell: bash

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::060378307146:role/github-actions-dea-notebooks--sandbox
aws-region: ap-southeast-2
role-duration-seconds: 7200 # 2 hours

- name: Get Database Credentials
run: |
username_password=$(aws ssm get-parameter --name /dea-sandbox-eks/sandbox_reader/db.creds --with-decryption --query Parameter.Value --output text)
# If we need separate username/password they can be split as follows.
# However exporting them to GitHub Environment variables will leave the Password in the Logs, if further steps aren't taken.
# IFS=: read -r PGUSERNAME PGPASSWORD <<< $username_password
# echo "PGUSERNAME=${PGUSERNAME}" >> $GITHUB_ENV
# echo "PGPASSWORD=${PGPASSWORD}" >> $GITHUB_ENV
echo DATACUBE_DB_URL=postgresql://${username_password}@localhost:5432/odc >> $GITHUB_ENV
shell: bash

- name: Open Port Forward to RDS
run: |
npx basti connect \
--custom-target-vpc vpc-086904199e505c1f6 \
--custom-target-host db-aurora-dea-sandbox-eks-1.cos5zfpkso9m.ap-southeast-2.rds.amazonaws.com \
--custom-target-port 5432 \
--local-port 5432 &
# Wait until the connection comes up, but, if it doesn't, don't hang forever.
npx wait-on --timeout 120000 --interval 1000 tcp:127.0.0.1:5432
echo "PGPORT=5432" >> $GITHUB_ENV
echo "PGHOST=localhost" >> $GITHUB_ENV
shell: bash
53 changes: 0 additions & 53 deletions .github/workflows/test_notebooks.yml

This file was deleted.

79 changes: 79 additions & 0 deletions .github/workflows/test_notebooks_pullrequest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@

name: Test changed files on pull request

on:
pull_request:
branches: [ develop ]

permissions:
id-token: write # Required for OIDC authentication to AWS
contents: read

jobs:
test-notebooks:
runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v4
with:
path: dea-notebooks

- name: Get changed notebook files
id: changed-notebooks
uses: tj-actions/changed-files@v45
with:
files: '**/*.ipynb'
separator: ' '

- name: Print changed notebook files
if: steps.changed-notebooks.outputs.any_changed == 'true'
run: |
echo "Changed notebook files:"
echo "${{ steps.changed-notebooks.outputs.all_changed_files }}"
- name: Set up database access and tests
if: steps.changed-notebooks.outputs.any_changed == 'true'
uses: ./dea-notebooks/.github/actions/test_setup
with:
image: stable

- name: Test changed Jupyter notebooks only
if: steps.changed-notebooks.outputs.any_changed == 'true'
run: |
sudo chown -R 1000:100 ./dea-notebooks
cd ./dea-notebooks
docker run --rm \
--net=host \
--env DATACUBE_DB_URL \
--env AWS_SESSION_TOKEN \
--env AWS_REGION \
--env AWS_ACCESS_KEY_ID \
--env AWS_SECRET_ACCESS_KEY \
--env AWS_SESSION_TOKEN \
--volume ${GITHUB_WORKSPACE}/dea-notebooks:/home/jovyan/dea-notebooks \
--volume ${GITHUB_WORKSPACE}/tide_models:/var/share/tide_models \
--env GDAL_HTTP_MAX_RETRY=10 \
--entrypoint /bin/bash \
538673716275.dkr.ecr.ap-southeast-2.amazonaws.com/geoscienceaustralia/sandbox:stable \
/home/jovyan/dea-notebooks/Tests/test_notebooks.sh "${{ steps.changed-notebooks.outputs.all_changed_files }}"
- name: Test DEA Tools
if: steps.changed-notebooks.outputs.any_changed == 'true'
run: |
sudo chown -R 1000:100 ./dea-notebooks
cd ./dea-notebooks
docker run --rm \
--net=host \
--env DATACUBE_DB_URL \
--env AWS_SESSION_TOKEN \
--env AWS_REGION \
--env AWS_ACCESS_KEY_ID \
--env AWS_SECRET_ACCESS_KEY \
--env AWS_SESSION_TOKEN \
--volume ${GITHUB_WORKSPACE}/dea-notebooks:/home/jovyan/dea-notebooks \
--volume ${GITHUB_WORKSPACE}/tide_models:/var/share/tide_models \
--env GDAL_HTTP_MAX_RETRY=10 \
--entrypoint /bin/bash \
538673716275.dkr.ecr.ap-southeast-2.amazonaws.com/geoscienceaustralia/sandbox:stable \
/home/jovyan/dea-notebooks/Tests/test_notebooks.sh dea_tools
65 changes: 65 additions & 0 deletions .github/workflows/test_notebooks_scheduled.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@

name: Test entire repo against DEA Sandbox images

on:

# Run on the hour
schedule:
- cron: "0 10 * * 0,3"

# Optional: Allow manual triggering through GitHub UI
workflow_dispatch:

permissions:
id-token: write # Required for OIDC authentication to AWS
contents: read

jobs:
test-notebooks:
runs-on: ubuntu-latest

# Test DEA Notebooks sections against both stable and latest Sandbox image
strategy:

matrix:
image:
- "latest"
- "stable"
tests:
- "dea_tools"
- "beginners_guide"
- "dea_products"
- "how_to_guides"
- "real_world_examples"
- "scalable_machine_learning"
fail-fast: false

steps:

- uses: actions/checkout@v4
with:
path: dea-notebooks

- name: Set up database access and tests
uses: ./dea-notebooks/.github/actions/test_setup
with:
image: ${{ matrix.image }}

- name: Run tests against each DEA Notebooks section
run: |
sudo chown -R 1000:100 ./dea-notebooks
cd ./dea-notebooks
docker run --rm \
--net=host \
--env DATACUBE_DB_URL \
--env AWS_SESSION_TOKEN \
--env AWS_REGION \
--env AWS_ACCESS_KEY_ID \
--env AWS_SECRET_ACCESS_KEY \
--env AWS_SESSION_TOKEN \
--volume ${GITHUB_WORKSPACE}/dea-notebooks:/home/jovyan/dea-notebooks \
--volume ${GITHUB_WORKSPACE}/tide_models:/var/share/tide_models \
--env GDAL_HTTP_MAX_RETRY=10 \
--entrypoint /bin/bash \
538673716275.dkr.ecr.ap-southeast-2.amazonaws.com/geoscienceaustralia/sandbox:${{ matrix.image }} \
/home/jovyan/dea-notebooks/Tests/test_notebooks.sh "${{ matrix.tests }}"
Loading

0 comments on commit 192d4d5

Please sign in to comment.