Merge pull request #64 from GeoscienceAustralia/metadata

Add new metadata folder to hold ODC product definition and metadata type

metadata/README.md

@@ -0,0 +1,16 @@
+# DEA Intertidal Open Data Cube metadata
+
+This directory contains metadata files required for indexing DEA Intertidal into the Open Data Cube (ODC), including:
+* An [ODC Product Definition YAML](https://datacube-core.readthedocs.io/en/latest/installation/product-definitions.html) describing the DEA Intertidal product and its bands/measurements
+* An [ODC Metadata Type YAML](https://datacube-core.readthedocs.io/en/latest/installation/metadata-types.html) defining custom searchable metadata fields for DEA Intertidal
+
+Individual [ODC Dataset Documents](https://datacube-core.readthedocs.io/en/latest/installation/dataset-documents.html) and Spatiotemporal Asset Catalogue (STAC) metadata are generated during the product generation workflow using the [`intertidal.io.export_dataset_metadata`](https://github.com/GeoscienceAustralia/dea-intertidal/blob/main/intertidal/io.py#L877-L1091) function.
+
+All three metadata files can be validated using the `eo3-validate` command from `eodatasets`:
+```
+!eo3-validate \
+metadata/ga_s2ls_intertidal_cyear_3.odc-product.yaml \
+metadata/eo3_intertidal.odc-type.yaml \
+data/processed/ga_s2ls_intertidal_cyear_3/0-0-1/tes/ting/2023--P1Y/ga_s2ls_intertidal_cyear_3_testing_2023--P1Y_interim.odc-metadata.yaml \
+--thorough
+```

metadata/eo3_intertidal.odc-type.yaml

@@ -0,0 +1,163 @@
+---
+name: eo3_intertidal
+description: EO3 for DEA Intertidal 
+dataset:
+  id: [id]
+  sources: [lineage, source_datasets]
+
+  grid_spatial: [grid_spatial, projection]
+  measurements: [measurements]
+  creation_dt: [properties, 'odc:processing_datetime']
+  label: [label]
+  format: [properties, 'odc:file_format']
+
+  search_fields:
+    platform:
+      description: Platform code
+      offset: [properties, 'eo:platform']
+      indexed: false
+
+    instrument:
+      description: Instrument name
+      offset: [properties, 'eo:instrument']
+      indexed: false
+
+    product_family:
+      description: Product family code
+      offset: [properties, 'odc:product_family']
+      indexed: false
+
+    region_code:
+      description: |
+        The unique GridSpec reference of the 32 km coastal analysis tile.  
+      offset: [properties, 'odc:region_code']
+
+    dataset_maturity:
+      description: One of - final|interim
+      offset: [properties, 'dea:dataset_maturity']
+
+    time:
+      description: Acquisition time range
+      type: datetime-range
+      min_offset:
+        - [properties, 'dtr:start_datetime']
+        - [properties, datetime]
+      max_offset:
+        - [properties, 'dtr:end_datetime']
+        - [properties, datetime]
+
+    lon:
+      description: Longitude range
+      type: double-range
+      min_offset:
+        - [extent, lon, begin]
+      max_offset:
+        - [extent, lon, end]
+
+    lat:
+      description: Latitude range
+      type: double-range
+      min_offset:
+        - [extent, lat, begin]
+      max_offset:
+        - [extent, lat, end]
+
+    eo_gsd:
+      description: Ground sample distance, meters
+      indexed: false
+      offset:
+        - properties
+        - eo:gsd
+      type: double
+
+    # Intertidal-specific metadata below
+    intertidal_category:
+      description: |
+        Tide range classification - one of microtidal|mesotidal|macrotidal
+      indexed: false
+      offset:
+        - properties
+        - intertidal:category
+
+    intertidal_hat:
+      description: |
+        Highest astronomical tide height (metres above Mean Sea Level)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:hat
+
+    intertidal_hot:
+      description: |
+        Highest satellite-observed tide height (metres above Mean Sea Level)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:hot
+
+    intertidal_lat:
+      description: |
+        Lowest astronomical tide height (metres above Mean Sea Level)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:lat
+
+    intertidal_lot:
+      description: |
+        Lowest satellite-observed tide height (metres above Mean Sea Level)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:lot
+
+    intertidal_offset_high:
+      description: |
+        Difference between highest astronomical and highest satellite-observed
+        tides (in percentage of tide range)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:offset_high
+
+    intertidal_offset_low:
+      description: |
+        Difference between lowest astronomical and lowest satellite-observed
+        tides (in percentage of tide range)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:offset_low
+
+    intertidal_spread:
+      description: |
+        Proportion of the tide range observed by satellites (percentage)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:spread
+
+    intertidal_tr:
+      description: |
+        Tide range (difference between highest and lowest astronomical tides)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:tr
+
+    intertidal_otr:
+      description: |
+        Observed tide range (difference between highest and lowest observed tides)
+      indexed: false
+      type: double
+      offset:
+        - properties
+        - intertidal:otr

data/raw/ga_s2ls_intertidal_cyear_3.odc-product.yaml → metadata/ga_s2ls_intertidal_cyear_3.odc-product.yaml

@@ -1,6 +1,6 @@
 ---
 name: ga_s2ls_intertidal_cyear_3
-metadata_type: eo3
+metadata_type: eo3_intertidal
 description: Geoscience Australia Sentinel-2 Landsat Intertidal Calendar Year Collection 3
 
 license: CC-BY-4.0

tests/README.md

@@ -10,7 +10,7 @@ Integration tests
 
 This directory contains tests that are run to verify that DEA Intertidal code runs correctly. The ``test_intertidal.py`` file runs a small-scale full workflow analysis over an intertidal flat in the Gulf of Carpentaria using the DEA Intertidal [Command Line Interface (CLI) tools](../notebooks/Intertidal_CLI.ipynb), and compares these results against a LiDAR validation DEM to produce some simple accuracy metrics.
 
-The latest integration test completed at **2024-02-29 15:26**. Compared to the previous run, it had an:
+The latest integration test completed at **2024-03-01 16:12**. Compared to the previous run, it had an:
 - RMSE accuracy of **0.14 m ( :heavy_minus_sign: no change)**
 - MAE accuracy of **0.12 m ( :heavy_minus_sign: no change)**
 - Bias of **0.12 m ( :heavy_minus_sign: no change)**

tests/test_intertidal.py

@@ -249,15 +249,16 @@ def test_dem_accuracy(
 @pytest.mark.dependency(depends=["test_intertidal_cli"])
 def test_validate_metadata():
     """
-    Validates output EO3 metadata against product definition.
+    Validates output EO3 metadata against product definition and metadata type.
     This will detect issues like incorrect datatypes, band names, nodata
     or missing bands.
     """
     runner = CliRunner()
     result = runner.invoke(
         eodatasets3.validate.run,
         [
-            "data/raw/ga_s2ls_intertidal_cyear_3.odc-product.yaml",
+            "metadata/ga_s2ls_intertidal_cyear_3.odc-product.yaml",
+            "metadata/eo3_intertidal.odc-type.yaml",
             "data/processed/ga_s2ls_intertidal_cyear_3/0-0-1/tes/ting/2021--P1Y/ga_s2ls_intertidal_cyear_3_testing_2021--P1Y_final.odc-metadata.yaml",            
             "--thorough",
         ],

tests/validation.csv

@@ -31,3 +31,4 @@ time,Correlation,RMSE,MAE,R-squared,Bias,Regression slope
 2024-02-29 03:11:40.503223+00:00,0.976,0.141,0.121,0.952,0.117,1.109
 2024-02-29 03:34:20.070327+00:00,0.976,0.141,0.121,0.952,0.117,1.109
 2024-02-29 04:26:05.610010+00:00,0.976,0.141,0.121,0.952,0.117,1.109
+2024-03-01 05:12:24.737271+00:00,0.976,0.141,0.121,0.952,0.117,1.109