Merge pull request #376 from GeoscienceAustralia/feature/quality

Copy tech alerts over to relevant product pages so they can be found

docs/data/product/dea-geometric-median-and-median-absolute-deviation-landsat/_quality.md

@@ -9,3 +9,12 @@ To reduce the impact of poor-quality earth observation data, the input data is s
 Note that where no-data pixels occur in locations that clear observations should occur, the cause is a lack of clear pixels to populate the GeoMAD. 
 
 Also, note that since  the geometric median and MAD are synthetic (not observed directly by satellite), they cannot be verified by field work.
+
+## Known Issues
+
+:::{dropdown} 19 Sep 2024: GeoMAD processing bug
+:open:
+
+In the [DEA Geometric Median and Median Absolute Deviation (Landsat)](/data/product/dea-geometric-median-and-median-absolute-deviation-landsat/) product, a bug in the processing code related to multithreading using numexpr (a fast numerical expression evaluator for Numpy) has been identified which may cause a 400x400 pixel data block to be misplaced within a tile. For the full GeoMAD archive, it is possible that there are around 8-12 tiles with incorrect data (misplaced blocks). It is unknown at this stage which tiles are affected. We are investigating the bug and will provide more information in late 2024 to early 2025.
+:::
+

docs/data/product/dea-hotspots/_data.yaml

@@ -101,7 +101,7 @@ enable_description: true
 enable_quality: true
 enable_specifications: true
 enable_access: true
-enable_history: false
+enable_history: true
 enable_faqs: false
 enable_credits: true
 

docs/data/product/dea-hotspots/_history.md

@@ -1,2 +1,16 @@
-% ## Changelog
+## Changelog
+
+### 19 Dec 2024: NOAA-21-derived DEA Hotspots are now available
+
+Following our [communication](https://communication.ga.gov.au/link/id/zzzz675f7ce74a008871Pzzzz6567c8b713b5b826/page.html) in November regarding changes to the Hotspots service, we are pleased to advise that Hotspots derived from the Visible Infrared Imaging Radiometer Suite (VIIRS) sensor onboard NOAA-21 are now available. NOAA-21 currently passes over Australia at approximately 3pm and 2am Australian Eastern Standard Time, increasing the density and reliability of the afternoon and nighttime observations.
+
+In the Hotspots application, the NOAA-21 feed is off by default. You can include either the VIIRS ‘AFIMG NOAA-21’, or ‘AFMOD NOAA-21’ Hotspots algorithms as additional layers in the Hotspots map interface by selecting these layers from the VIIRS menu in the legend.  
+
+[View the Tech Alert](https://communication.ga.gov.au/DEAHotspots-19Dec24)
+
+### 18 Nov 2024: DEA Hotspots — Enhancing Bushfire Monitoring with New Satellite Data
+
+We are writing to keep you informed of changes that are occurring to the DEA Hotspots service. While faced with the loss of the MODIS Terra for our morning observations, we can continue to use available satellites to bolster the density and reliability of our afternoon observations. Data from the Visible Infrared Imaging Radiometer Suite (VIIRS) sensor onboard NOAA-21 will be available in DEA Hotspots by early December 2024. Once NOAA-21 data is available, a tech alert will be issued to users.
+
+[View the Tech Alert](https://communication.ga.gov.au/DEA-Hotspots-18Nov)
 

docs/data/product/dea-hotspots/_quality.md

@@ -24,3 +24,12 @@ Hotspots may be obscured or missing from the map due to the following reasons:
 * Sensors can be inoperable for extended periods of time, disrupting the detection of Hotspots.  
 * The fire may have been burning during a time when no satellite was looking over the fire ground.  
 
+## Known Issues
+
+:::{dropdown} 13 May 2024: Terra-derived DEA Hotspots are unavailable
+:open:
+
+Direct Broadcast satellite downloads from the Terra satellite have again become unavailable. This means that Terra-derived [DEA Hotspots](https://hotspots.dea.ga.gov.au/) are unavailable until further notice.
+
+This is due to the TERRA MODIS satellite experiencing power problems. The satellite continues to collect data but its direct broadcast has stopped.
+:::

docs/data/product/dea-surface-reflectance-nbart-sentinel-2a-msi/_quality.md

@@ -25,3 +25,83 @@ The results for each collection were averaged and then compared. The comparison
 
 The technical report containing the data summary for the Phase 1 DEA Surface Reflectance Validation is available: [DEA Analysis Ready Data Phase 1 Validation Project: Data Summary](http://pid.geoscience.gov.au/dataset/ga/145101)
 
+## Known Issues
+
+:::{dropdown} 26 Sep 2024: 's2cloudless' ARD reprocessing underway
+:open:
+
+We have begun reprocessing the older Sentinel-2 ARD data that was impacted by a bug in the s2cloudless cloud masking layer. The bug is '[24 May 2024: Misclassification issue with Sentinel-2 ‘s2cloudless’ cloud masking from 2022](#may-2024-misclassification-issue-with-sentinel-2-s2cloudless-cloud-masking-from-2022)'.
+
+The addition of an offset factor in ESA's Sentinel-2 L1C Processing Baseline 4.0.0 [on January 25 2022](https://sentiwiki.copernicus.eu/web/s2-processing) led to the generation of incorrect `s2cloudless` cloud classifications in our systems. This resulted in an over-classification of cloud, particularly over bare and agricultural regions.
+
+Sentinel-2 `s2cloudless` data from 25 January 2022 to 7 June 2024 is affected, and we expect to replace these datasets over the next several months.
+
+In the meantime, we advise the users to avoid using `s2cloudless` for cloud masking on data between these two dates, and consider using the `Fmask` cloud mask as a temporary alternative during this period.
+:::
+
+:::{dropdown} 24 May 2024: Misclassification issue with Sentinel-2 's2cloudless' cloud masking from 2022
+:open:
+
+An issue has been identified that is causing widespread misclassification of clouds in DEA's Sentinel-2 `s2cloudless` cloud mask data generated since January 2022. 
+
+We recommend that you avoid using `s2cloudless` cloud mask data from 2022 onwards until this issue is investigated. 
+:::
+
+:::{dropdown} 25 Oct 2023: Removing Sentinel-2 Level 1 and ARD duplicates (Resolved)
+
+Duplicate Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) has been discovered. The duplicate groups have the same region code and datetime attribute.
+
+**Background**
+
+Remediation will require archiving the duplicate scenes of our Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) products.
+
+Which Products are affected?
+
+* S2b
+* S2a
+
+**What does this mean for DEA Users?**
+
+This is expected to have minimal impact on users.
+
+If you have any further questions or encounter issues, please contact <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/link/id/zzzz65384bbe2a28c901Pzzzz61de67bd94bfe861/page.html)
+:::
+
+:::{dropdown} Aug 2023: Sentinel-2 ARD GQA coastal scenes error; DEA Burnt Area (S2 NRT, Provisional) decommissioning (Resolved)
+
+**Sentinel-2 ARD error &mdash; GQA failing over coastal scenes**
+
+Many Sentinel-2 Analysis Ready Data (ARD) coastal scenes up until March 2023 were produced without geometric quality assessment (GQA) information.
+
+This error was found and fixed in the ARD processing code, however there are data errors present within existing coastal ARD produced prior to March 2023.
+
+**Background**
+
+GQA (Geometric Quality Assessment) metadata provides information about the georeferencing accuracy of each satellite observation.
+
+Sentinel-2 ARD coastal scenes produced before March 2023 are missing GQA information, leading to low quality ARD outputs.
+
+To correct the issue, we plan to archive and remove 26,037 Sentinel-2 ARD coastal scenes that have no GQA data.
+
+Archiving and removing the scenes with no GQA will lead to good ARD being produced.
+
+**Which Products are affected?**
+
+* 'ga_s2am_ard_3'
+* 'ga_s2bm_ard_3'
+* Affected scene regions: '56JNR', '53HMC', '52HDK', '50KQD', '50HQG', '52LEL', '55GDM', '54HUE', '56HLG', '51HWC', '53HKE', '52HCK', '56JNP', '50HLG', '52LCK', '53LQG', '52LFM', '52HGK', '55GBQ', '54HVC', '56JNQ', '55HFU', '49KHS', '55HGU', '50HNG', '50HPG', '53LLH', '56HLH', '55GEM', '50HMG', '52LEK', '54HUD'
+
+**What does this mean for DEA Users?**
+
+Coastal ARD scenes before March 2023 that have no GQA data will be temporarily unavailable. They will be replaced with new ARD scenes with GQA data.
+
+Users who have previously accessed Sentinel-2 ARD over coastal areas and filtered by GQA may wish to re-run their workflows to account for the updated data.
+
+If you have any questions about the above, please email us at <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/pub/pubType/EO/pubID/zzzz64dc2b594744b162/interface.html)
+
+:::
+

docs/data/product/dea-surface-reflectance-nbart-sentinel-2b-msi/_quality.md

@@ -25,3 +25,82 @@ The results for each collection were averaged and then compared. The comparison
 
 The technical report containing the data summary for the Phase 1 DEA Surface Reflectance Validation is available: [DEA Analysis Ready Data Phase 1 Validation Project: Data Summary](http://pid.geoscience.gov.au/dataset/ga/145101)
 
+## Known Issues
+
+:::{dropdown} 26 Sep 2024: 's2cloudless' ARD reprocessing underway
+:open:
+
+We have begun reprocessing the older Sentinel-2 ARD data that was impacted by a bug in the s2cloudless cloud masking layer. The bug is '[24 May 2024: Misclassification issue with Sentinel-2 ‘s2cloudless’ cloud masking from 2022](#may-2024-misclassification-issue-with-sentinel-2-s2cloudless-cloud-masking-from-2022)'.
+
+The addition of an offset factor in ESA's Sentinel-2 L1C Processing Baseline 4.0.0 [on January 25 2022](https://sentiwiki.copernicus.eu/web/s2-processing) led to the generation of incorrect `s2cloudless` cloud classifications in our systems. This resulted in an over-classification of cloud, particularly over bare and agricultural regions.
+
+Sentinel-2 `s2cloudless` data from 25 January 2022 to 7 June 2024 is affected, and we expect to replace these datasets over the next several months.
+
+In the meantime, we advise the users to avoid using `s2cloudless` for cloud masking on data between these two dates, and consider using the `Fmask` cloud mask as a temporary alternative during this period.
+:::
+
+:::{dropdown} 24 May 2024: Misclassification issue with Sentinel-2 's2cloudless' cloud masking from 2022
+:open:
+
+An issue has been identified that is causing widespread misclassification of clouds in DEA's Sentinel-2 `s2cloudless` cloud mask data generated since January 2022. 
+
+We recommend that you avoid using `s2cloudless` cloud mask data from 2022 onwards until this issue is investigated. 
+:::
+
+:::{dropdown} 25 Oct 2023: Removing Sentinel-2 Level 1 and ARD duplicates (Resolved)
+
+Duplicate Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) has been discovered. The duplicate groups have the same region code and datetime attribute.
+
+**Background**
+
+Remediation will require archiving the duplicate scenes of our Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) products.
+
+Which Products are affected?
+
+* S2b
+* S2a
+
+**What does this mean for DEA Users?**
+
+This is expected to have minimal impact on users.
+
+If you have any further questions or encounter issues, please contact <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/link/id/zzzz65384bbe2a28c901Pzzzz61de67bd94bfe861/page.html)
+:::
+
+:::{dropdown} Aug 2023: Sentinel-2 ARD GQA coastal scenes error; DEA Burnt Area (S2 NRT, Provisional) decommissioning (Resolved)
+
+**Sentinel-2 ARD error &mdash; GQA failing over coastal scenes**
+
+Many Sentinel-2 Analysis Ready Data (ARD) coastal scenes up until March 2023 were produced without geometric quality assessment (GQA) information.
+
+This error was found and fixed in the ARD processing code, however there are data errors present within existing coastal ARD produced prior to March 2023.
+
+**Background**
+
+GQA (Geometric Quality Assessment) metadata provides information about the georeferencing accuracy of each satellite observation.
+
+Sentinel-2 ARD coastal scenes produced before March 2023 are missing GQA information, leading to low quality ARD outputs.
+
+To correct the issue, we plan to archive and remove 26,037 Sentinel-2 ARD coastal scenes that have no GQA data.
+
+Archiving and removing the scenes with no GQA will lead to good ARD being produced.
+
+**Which Products are affected?**
+
+* 'ga_s2am_ard_3'
+* 'ga_s2bm_ard_3'
+* Affected scene regions: '56JNR', '53HMC', '52HDK', '50KQD', '50HQG', '52LEL', '55GDM', '54HUE', '56HLG', '51HWC', '53HKE', '52HCK', '56JNP', '50HLG', '52LCK', '53LQG', '52LFM', '52HGK', '55GBQ', '54HVC', '56JNQ', '55HFU', '49KHS', '55HGU', '50HNG', '50HPG', '53LLH', '56HLH', '55GEM', '50HMG', '52LEK', '54HUD'
+
+**What does this mean for DEA Users?**
+
+Coastal ARD scenes before March 2023 that have no GQA data will be temporarily unavailable. They will be replaced with new ARD scenes with GQA data.
+
+Users who have previously accessed Sentinel-2 ARD over coastal areas and filtered by GQA may wish to re-run their workflows to account for the updated data.
+
+If you have any questions about the above, please email us at <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/pub/pubType/EO/pubID/zzzz64dc2b594744b162/interface.html)
+:::
+

docs/data/product/dea-surface-reflectance-oa-sentinel-2a-msi/_quality.md

@@ -26,3 +26,82 @@ The first Cloud Mask Intercomparison eXercise (CMIX) validated the Fmask and the
 
 The calculation of the satellite and solar positional geometry datasets are largely influenced by the publicly available ephemeris data and whether the satellite has an on-board GPS, as well as the geographical information that resides with the imagery data and the metadata published by the data providers. The code to generate the geometry grids is routinely tested and evaluated for accuracy at >6 decimal places of precision.
 
+## Known Issues
+
+:::{dropdown} 26 Sep 2024: 's2cloudless' ARD reprocessing underway
+:open:
+
+We have begun reprocessing the older Sentinel-2 ARD data that was impacted by a bug in the s2cloudless cloud masking layer. The bug is '[24 May 2024: Misclassification issue with Sentinel-2 ‘s2cloudless’ cloud masking from 2022](#may-2024-misclassification-issue-with-sentinel-2-s2cloudless-cloud-masking-from-2022)'.
+
+The addition of an offset factor in ESA's Sentinel-2 L1C Processing Baseline 4.0.0 [on January 25 2022](https://sentiwiki.copernicus.eu/web/s2-processing) led to the generation of incorrect `s2cloudless` cloud classifications in our systems. This resulted in an over-classification of cloud, particularly over bare and agricultural regions.
+
+Sentinel-2 `s2cloudless` data from 25 January 2022 to 7 June 2024 is affected, and we expect to replace these datasets over the next several months.
+
+In the meantime, we advise the users to avoid using `s2cloudless` for cloud masking on data between these two dates, and consider using the `Fmask` cloud mask as a temporary alternative during this period.
+:::
+
+:::{dropdown} 24 May 2024: Misclassification issue with Sentinel-2 's2cloudless' cloud masking from 2022
+:open:
+
+An issue has been identified that is causing widespread misclassification of clouds in DEA's Sentinel-2 `s2cloudless` cloud mask data generated since January 2022. 
+
+We recommend that you avoid using `s2cloudless` cloud mask data from 2022 onwards until this issue is investigated. 
+:::
+
+:::{dropdown} 25 Oct 2023: Removing Sentinel-2 Level 1 and ARD duplicates (Resolved)
+
+Duplicate Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) has been discovered. The duplicate groups have the same region code and datetime attribute.
+
+**Background**
+
+Remediation will require archiving the duplicate scenes of our Sentinel-2 Level 1 (s2 l1) and Analysis Ready Data (ARD) products.
+
+Which Products are affected?
+
+* S2b
+* S2a
+
+**What does this mean for DEA Users?**
+
+This is expected to have minimal impact on users.
+
+If you have any further questions or encounter issues, please contact <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/link/id/zzzz65384bbe2a28c901Pzzzz61de67bd94bfe861/page.html)
+:::
+
+:::{dropdown} Aug 2023: Sentinel-2 ARD GQA coastal scenes error; DEA Burnt Area (S2 NRT, Provisional) decommissioning (Resolved)
+
+**Sentinel-2 ARD error &mdash; GQA failing over coastal scenes**
+
+Many Sentinel-2 Analysis Ready Data (ARD) coastal scenes up until March 2023 were produced without geometric quality assessment (GQA) information.
+
+This error was found and fixed in the ARD processing code, however there are data errors present within existing coastal ARD produced prior to March 2023.
+
+**Background**
+
+GQA (Geometric Quality Assessment) metadata provides information about the georeferencing accuracy of each satellite observation.
+
+Sentinel-2 ARD coastal scenes produced before March 2023 are missing GQA information, leading to low quality ARD outputs.
+
+To correct the issue, we plan to archive and remove 26,037 Sentinel-2 ARD coastal scenes that have no GQA data.
+
+Archiving and removing the scenes with no GQA will lead to good ARD being produced.
+
+**Which Products are affected?**
+
+* 'ga_s2am_ard_3'
+* 'ga_s2bm_ard_3'
+* Affected scene regions: '56JNR', '53HMC', '52HDK', '50KQD', '50HQG', '52LEL', '55GDM', '54HUE', '56HLG', '51HWC', '53HKE', '52HCK', '56JNP', '50HLG', '52LCK', '53LQG', '52LFM', '52HGK', '55GBQ', '54HVC', '56JNQ', '55HFU', '49KHS', '55HGU', '50HNG', '50HPG', '53LLH', '56HLH', '55GEM', '50HMG', '52LEK', '54HUD'
+
+**What does this mean for DEA Users?**
+
+Coastal ARD scenes before March 2023 that have no GQA data will be temporarily unavailable. They will be replaced with new ARD scenes with GQA data.
+
+Users who have previously accessed Sentinel-2 ARD over coastal areas and filtered by GQA may wish to re-run their workflows to account for the updated data.
+
+If you have any questions about the above, please email us at <[email protected]>
+
+[View the Tech Alert](https://communication.ga.gov.au/pub/pubType/EO/pubID/zzzz64dc2b594744b162/interface.html)
+:::
+