Publish recent changes to stable branch (#1234)

* Add updated `xr_regression` function for multi-dimensional linear regression (#1226) * Add updated xr_regression function * Add dask support for lazy computation * Set dtypes * Update docstring * Update docstring * Add MAD outliers * Update docstring * Remove lag functionality * Update docstrings * Add better error handling * Update stream gauge corr notebook to use new func * Adding DEAfrica Wetland Turbidity notebook for Australian study site (#1175) * Adding DEAfrica Wetland Turbidity notebook for Australian study site * change all instances of NDTI to NDTI2 to reflect usage at top of notebook * update notebook to use Collection 3 WO Statistics * rerun notebook --------- Co-authored-by: BexDunn <[email protected]> * Add spatial interpolation with `xr_interpolate` notebook (#1233) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook * Remove coastal files from branch * Add points data * Review feedback; * Add p param to IDW * Fix test * Updates to product notebook Knowledge Hub links and DEA notebook content (#1221) * Move KH links into consistent alert box format * Update DEA notebook * Minor wording updates * Minor wording * Temporarily remove STAC notebook from tests * Add ensemble tide modelling functionality to `model_tides` (#1231) * Add ensemble tide modelling functionality to model_tides * Update test_coastal.py * Remove test * Updates to IDW, xr_interpolate and ensemble tide modelling code" * Doco updates * Switch ensemble rankings from high to low = good * Update docstring * Fix doco * Add interpolation notebook --------- Co-authored-by: Matt-dea <[email protected]> Co-authored-by: BexDunn <[email protected]>
GeoscienceAustralia · Jun 17, 2024 · 9412d77 · 9412d77
1 parent 58097d3
commit 9412d77
Show file tree

Hide file tree

Showing 25 changed files with 4,318 additions and 1,237 deletions.
diff --git a/Beginners_guide/02_DEA.ipynb b/Beginners_guide/02_DEA.ipynb
@@ -83,9 +83,10 @@
     "The earliest datasets of optical satellite imagery in DEA date from 1986.\n",
     "DEA includes data from:\n",
     "\n",
-    "* [Landsat 5 TM](https://www.usgs.gov/land-resources/nli/landsat/landsat-5?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS5 TM), operational between March 1984 and January 2013\n",
-    "* [Landsat 7 ETM+](https://www.usgs.gov/land-resources/nli/landsat/landsat-7?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS7 ETM+), operational since April 1999\n",
-    "* [Landsat 8 OLI](https://www.usgs.gov/land-resources/nli/landsat/landsat-8?qt-science_support_page_related_con=0#qt-science_support_page_related_con) (LS8 OLI), operational since February 2013\n",
+    "* [Landsat 5 TM](https://www.usgs.gov/landsat-missions/landsat-5) (LS5 TM), operational between March 1984 and January 2013\n",
+    "* [Landsat 7 ETM+](https://www.usgs.gov/landsat-missions/landsat-7) (LS7 ETM+), operational between April 1999 and April 2022\n",
+    "* [Landsat 8 OLI](https://www.usgs.gov/landsat-missions/landsat-8) (LS8 OLI), operational since February 2013\n",
+    "* [Landsat 9 OLI](https://www.usgs.gov/landsat-missions/landsat-9) (LS9 OLI), operational since September 2021\n",
     "* [Sentinel 2A MSI](https://sentinel.esa.int/web/sentinel/missions/sentinel-2) (S2A MSI), operational since June 2015\n",
     "* [Sentinel 2B MSI](https://sentinel.esa.int/web/sentinel/missions/sentinel-2) (S2B MSI, operational since March 2017\n",
     "\n",
@@ -248,7 +249,7 @@
     "\n",
     "By default, the spatial extent of the DEA data holdings is approximately the Australian coastal shelf. \n",
     "The actual extent varies based on the sensor and product. \n",
-    "The current extents of each DEA product can be viewed using the interactive [DEA Datacube Explorer](https://explorer.dea.ga.gov.au)."
+    "The current extents of each DEA product can be viewed using the interactive [DEA Explorer](https://explorer.dea.ga.gov.au)."
    ]
   },
   {
@@ -257,32 +258,20 @@
    "source": [
     "## Derived products\n",
     "\n",
-    "![DEA products](../Supplementary_data/02_DEA/dea_products.jpg)\n",
+    "![](../Supplementary_data/02_DEA/dea_products.jpg)\n",
     "\n",
     "In addition to ARD satellite data, DEA generates a range of products that are derived from Landsat or Sentinel-2 surface reflectance data.\n",
     "These products have been developed to characterise and monitor different aspects of Australia's natural and built environment, such as mapping the distribution of water and vegetation across the landscape through time.\n",
-    "Derived DEA products include:\n",
     "\n",
-    "* **Water Observations from Space (WOfS):** WOfS is the world's first continent-scale map of surface water and provides images and data showing where water has been seen in Australia from 1987 to the present. This map can be used to better understand where water usually occurs across the continent and to plan water management strategies. \n",
-    "\n",
-    "* **Fractional Cover (FC):** Fractional Cover (FC) is a measurement that splits the landscape into three parts, or fractions; green (leaves, grass, and growing crops), brown (branches, dry grass or hay, and dead leaf litter), and bare ground (soil or rock). DEA uses Fractional Cover to characterise every 25 m square of Australia for any point in time from 1987 to today. This measurement can inform a broad range of natural resource management issues. \n",
-    "\n",
-    "* **High and Low Tide Composites (HLTC):** The High and Low Tide Composites (HLTC) are imagery mosaics developed to visualise Australia's coasts, estuaries and reefs at low and high tide, whilst removing the influence of noise features such as clouds, breaking water and sun-glint. These products are highly interpretable, and provide a valuable snapshot of the coastline at different biophysical states.\n",
-    "\n",
-    "* **Intertidal Extents Model (ITEM):** The Intertidal Extents Model (ITEM) product utilises 30 years of Earth observation data from the Landsat archive to map the extents and topography of Australia's intertidal mudflats, beaches and reefs; the area exposed between high and low tide.\n",
-    "\n",
-    "* **National Intertidal Digital Elevation Model (NIDEM):** The National Intertidal Digital Elevation Model (NIDEM) is a national dataset that maps the three-dimensional structure of Australia’s intertidal zone. NIDEM provides a first-of-its kind source of intertidal elevation data for Australia’s entire coastline. \n",
-    "\n",
-    "Each of the products above have dataset-specific naming conventions, measurements, resolutions, data types and coordinate reference systems.\n",
-    "For more information about DEA's derived products, refer to the [DEA website](http://www.ga.gov.au/dea/products), the [Content Management Interface](https://knowledge.dea.ga.gov.au/) (CMI) containing detailed product metadata, or the \"DEA_products\" notebooks in this repository."
+    "For more information about DEA's derived products, refer to the [DEA website](http://www.ga.gov.au/dea/products), the [Data Products page of the DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/), or the \"DEA_products\" notebooks on the DEA Sandbox (e.g. [Introduction to DEA Surface Reflectance (Landsat, Collection 3)](../DEA_products/DEA_Landsat_Surface_Reflectance.ipynb))."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Recommended next steps\n",
-    "For more detailed information on the concepts introduced in this notebook, please see the [DEA User Guide](https://knowledge.dea.ga.gov.au/) and [Open Data Cube Manual](https://datacube-core.readthedocs.io/en/latest/).\n",
+    "For more detailed information on the concepts introduced in this notebook, please see the [DEA Knowledge Hub](https://knowledge.dea.ga.gov.au/) and [Open Data Cube Manual](https://datacube-core.readthedocs.io/en/latest/).\n",
     "For more information on the development of the DEA platform, please see [Dhu et al. 2017](https://doi.org/10.1080/20964471.2017.1402490).\n",
     "\n",
     "To continue with the beginner's guide, the following notebooks are designed to be worked through in the following order:\n",
@@ -319,7 +308,7 @@
     "**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).\n",
     "If you would like to report an issue with this notebook, you can file one on [GitHub](https://github.com/GeoscienceAustralia/dea-notebooks).\n",
     "\n",
-    "**Last modified:** December 2023"
+    "**Last modified:** April 2024"
    ]
   },
   {

diff --git a/DEA_products/DEA_Coastlines.ipynb b/DEA_products/DEA_Coastlines.ipynb
@@ -37,9 +37,6 @@
     "* Nanson, R., Bishop-Taylor, R., Sagar, S., Lymburner, L., (2022). Geomorphic insights into Australia's coastal change using a national dataset derived from the multi-decadal Landsat archive. Estuarine, Coastal and Shelf Science, 265, p.107712. Available: https://doi.org/10.1016/j.ecss.2021.107712\n",
     "* Bishop-Taylor, R., Sagar, S., Lymburner, L., Alam, I., & Sixsmith, J. (2019). Sub-pixel waterline extraction: Characterising accuracy and sensitivity to indices and spectra. *Remote Sensing*, 11(24), 2984. Available: https://www.mdpi.com/2072-4292/11/24/2984\n",
     "\n",
-    "> **Note:** For more technical information about the DEA Coastlines product, visit the official Geoscience Australia [DEA Coastlines product description](https://knowledge.dea.ga.gov.au/data/product/dea-coastlines)..\n",
-    "> To explore DEA Coastlines on an interactive map, [visit DEA Maps](https://maps.dea.ga.gov.au/story/DEACoastlines).\n",
-    "\n",
     "## Description\n",
     "\n",
     "This notebook will demonstrate how to load data from the [Digital Earth Australia Coastlines](https://knowledge.dea.ga.gov.au/data/product/dea-coastlines) product using the Digital Earth Australia datacube. \n",
@@ -48,6 +45,13 @@
     "1. Loading DEA Coastlines annual shoreline data using the `get_coastlines` function.\n",
     "2. Loading DEA Coastlines rates of change statistics data using the `get_coastlines` function.\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA Coastlines product documentation](https://knowledge.dea.ga.gov.au/data/product/dea-coastlines) for detailed technical information including methods, quality, and data access.\n",
+    "To explore DEA Coastlines on an interactive map, [visit DEA Maps](https://maps.dea.ga.gov.au/story/DEACoastlines).\n",
+    "    \n",
+    "</div>\n",
+    "\n",
     "***\n"
    ]
   },
@@ -1240,7 +1244,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

diff --git a/DEA_products/DEA_Fractional_Cover.ipynb b/DEA_products/DEA_Fractional_Cover.ipynb
@@ -40,9 +40,7 @@
     "* vegetation studies\n",
     "* fuel load estimation\n",
     "* ecosystem modelling\n",
-    "* land cover mapping\n",
-    "\n",
-    "> **Note:** For more technical information about DEA Fractional Cover, visit the official [Geoscience Australia DEA Fractional Cover product description](https://knowledge.dea.ga.gov.au/data/product/dea-fractional-cover-landsat)."
+    "* land cover mapping\n"
    ]
   },
   {
@@ -59,6 +57,12 @@
     "4. Inspecting unmixing error outputs\n",
     "4. Masking out missing or invalid data and unclear or wet pixels, and using this to track percentages of green and brown vegetation and bare soil over time\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA Fractional Cover product documentation](https://knowledge.dea.ga.gov.au/data/product/dea-fractional-cover-landsat) for detailed technical information including methods, quality, and data access.\n",
+    "    \n",
+    "</div>\n",
+    "\n",
     "***"
    ]
   },
@@ -1761,7 +1765,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

diff --git a/DEA_products/DEA_High_and_Low_Tide_Imagery.ipynb b/DEA_products/DEA_High_and_Low_Tide_Imagery.ipynb
@@ -36,9 +36,7 @@
     "* Visualising the full observed extent of the tidal range around the Australian continental coastline\n",
     "\n",
     "### Publications\n",
-    "* Sagar, S., Phillips, C., Bala, B., Roberts, D., & Lymburner, L. (2018). [Generating continental scale pixel-based surface reflectance composites in coastal regions with the use of a multi-resolution tidal model](https://www.mdpi.com/2072-4292/10/3/480). Remote Sensing, 10(3), 480.\n",
-    "\n",
-    "> **Note:** For more technical information about DEA High and Low Tide Imagery, visit the official [Geoscience Australia DEA High and Low Tide Imagery product description](https://knowledge.dea.ga.gov.au/data/product/dea-high-and-low-tide-imagery-landsat)."
+    "* Sagar, S., Phillips, C., Bala, B., Roberts, D., & Lymburner, L. (2018). [Generating continental scale pixel-based surface reflectance composites in coastal regions with the use of a multi-resolution tidal model](https://www.mdpi.com/2072-4292/10/3/480). Remote Sensing, 10(3), 480.\n"
    ]
   },
   {
@@ -54,6 +52,12 @@
     "3. Plotting low and high tide data in true and false colour\n",
     "4. Converting low and high tide data to a remote sensing water index (NDWI), and use this to map wet pixels at low and high tide\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA High and Low Tide Imagery product documentation](https://knowledge.dea.ga.gov.au/data/product/dea-high-and-low-tide-imagery-landsat) for detailed technical information including methods, quality, and data access.\n",
+    "\n",
+    "</div>\n",
+    "\n",
     "***"
    ]
   },
@@ -1106,7 +1110,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -1120,7 +1124,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

diff --git a/DEA_products/DEA_Intertidal.ipynb b/DEA_products/DEA_Intertidal.ipynb
diff --git a/DEA_products/DEA_Land_Cover.ipynb b/DEA_products/DEA_Land_Cover.ipynb
@@ -51,9 +51,7 @@
     "### Publications\n",
     "\n",
     "* Lucas R, Mueller N, Siggins A, Owers C, Clewley D, Bunting P, Kooymans C, Tissott B, Lewis B, Lymburner L, Metternicht G (2019) 'Land Cover Mapping using Digital Earth Australia', <em>Data</em>, 4(4):143, doi: [10.3390/data4040143](https://www.mdpi.com/2306-5729/4/4/143/htm).\n",
-    "* Owers C, Lucas R, Clewley D, Planque C, Punalekar S, Tissott B, Chua S, Bunting P, Mueller N, Metternicht G (2021) 'Living Earth: Implementing national standardised land cover classification systems for Earth Observation in support of sustainable development', <em>Big Earth Data</em>, 5(3):368-390, doi:[10.1080/20964471.2021.1948179](https://www.tandfonline.com/doi/full/10.1080/20964471.2021.1948179?scroll=top&needAccess=true).\n",
-    "\n",
-    "> **Note:** For more technical information about DEA Land Cover, visit the official Geoscience Australia [DEA Land Cover product description](https://knowledge.dea.ga.gov.au/data/product/dea-land-cover-landsat) page."
+    "* Owers C, Lucas R, Clewley D, Planque C, Punalekar S, Tissott B, Chua S, Bunting P, Mueller N, Metternicht G (2021) 'Living Earth: Implementing national standardised land cover classification systems for Earth Observation in support of sustainable development', <em>Big Earth Data</em>, 5(3):368-390, doi:[10.1080/20964471.2021.1948179](https://www.tandfonline.com/doi/full/10.1080/20964471.2021.1948179?scroll=top&needAccess=true).\n"
    ]
   },
   {
@@ -69,6 +67,13 @@
     "2. Choosing and loading DEA Land Cover data for an area of interest.\n",
     "3. Plotting DEA Land Cover data.\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA Land Cover product documentation](https://knowledge.dea.ga.gov.au/data/product/dea-land-cover-landsat) for detailed technical information including methods, quality, and data access.\n",
+    "To explore DEA Land Cover on an interactive map, [visit DEA Maps](https://maps.dea.ga.gov.au/story/DEALandCover).\n",
+    "\n",
+    "</div>\n",
+    "\n",
     "***"
    ]
   },
@@ -1873,7 +1878,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

diff --git a/DEA_products/DEA_Landsat_Surface_Reflectance.ipynb b/DEA_products/DEA_Landsat_Surface_Reflectance.ipynb
@@ -52,8 +52,6 @@
     "* Li, F., Jupp, D. L. B., Reddy, S., Lymburner, L., Mueller, N., Tan, P., & Islam, A. (2010). An evaluation of the use of atmospheric and BRDF correction to standardize Landsat data. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 3(3), 257–270. https://doi.org/10.1109/JSTARS.2010.2042281\n",
     "* Li, F., Jupp, D. L. B., Thankappan, M., Lymburner, L., Mueller, N., Lewis, A., & Held, A. (2012). A physics-based atmospheric and BRDF correction for Landsat data over mountainous terrain. *Remote Sensing of Environment*, 124, 756–770. https://doi.org/10.1016/j.rse.2012.06.018\n",
     "\n",
-    "> **Note:** For more technical information about the DEA Landsat Surface Reflectance products, visit the official Geoscience Australia [DEA Landsat Surface Reflectance product listings](https://knowledge.dea.ga.gov.au/data-products?combine=&program%5Bdea%5D=dea&catalog%5B12884%5D=12884).\n",
-    "\n",
     "## Description\n",
     "\n",
     "This notebook will demonstrate how to load data from the DEA Landsat Surface Reflectance products using the Digital Earth Australia datacube. \n",
@@ -66,6 +64,12 @@
     "5. [Advanced: Loading Landsat data in its native projection and resolution](#Native-load)\n",
     "6. [Advanced: Filtering Landsat data by product metadata](#Filtering-by-metadata)\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA Surface Reflectance product documentation](https://knowledge.dea.ga.gov.au/data/category/dea-surface-reflectance/) for detailed technical information including methods, quality, and data access.\n",
+    "\n",
+    "</div>\n",
+    "\n",
     "***\n"
    ]
   },
@@ -1900,7 +1904,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {

diff --git a/DEA_products/DEA_Mangroves.ipynb b/DEA_products/DEA_Mangroves.ipynb
@@ -54,12 +54,13 @@
   {
    "cell_type": "markdown",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
     "### Publications\n",
     "\n",
-    "[Lymburner, L., Bunting, P., Lucas, R., Scarth, P., Alam, I., Phillips, C., Ticehurst, C., & Held, A., (2020). Mapping the multi-decadal mangrove dynamics of the Australian coastline. Remote Sensing of Environment, 238, 111185.](https://doi.org/10.1016/j.rse.2019.05.004)"
+    "* Lymburner, L., Bunting, P., Lucas, R., Scarth, P., Alam, I., Phillips, C., Ticehurst, C., & Held, A., (2020). Mapping the multi-decadal mangrove dynamics of the Australian coastline. Remote Sensing of Environment, 238, 111185. Available at: https://doi.org/10.1016/j.rse.2019.05.004"
    ]
   },
   {
@@ -75,7 +76,15 @@
     "3. Plot a single timestep image\n",
     "4. Create and view an animation of the whole timeseries\n",
     "5. Plot change over time by graphing the timeseries of each class\n",
-    "6. Identify hotspot change areas within each class"
+    "6. Identify hotspot change areas within each class\n",
+    "\n",
+    "<div class=\"alert alert-info\">\n",
+    "    \n",
+    "**Note:** Visit the [DEA Mangroves product documentation](https://knowledge.dea.ga.gov.au/data/product/dea-mangrove-canopy-cover-landsat) for detailed technical information including methods, quality, and data access.\n",
+    "    \n",
+    "</div>\n",
+    "\n",
+    "***"
    ]
   },
   {
@@ -1459,7 +1468,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.13"
   },
   "widgets": {
    "application/vnd.jupyter.widget-state+json": {