Test merge to resolve conflicts (#470)

* Initial commit for development branch

- moved all notebooks (NCI and Sandbox) into Original_Structure folder

* Add new folder structure

- each folder has an empty readme, which should be updated with a description for the folder

* Directory restructure (#336)

* Change folders to have simple names and update README

* Change README files from .md to .rst for future use with documentation

* Change references from github.io page to docs.dea page in main README

* Add template to DEA sandbox repo (#337)

* Add descriptions to each folder's README.rst file (#340)

* Update notebook template (#338)

* Move additional information to end of notebook

- add information on highlighting parameters to getting started section

- add guideline around using subheadings

- break up markdown paragraphs so each sentence is on a new line to aid git-diff in future

- add example packages to load, including sourcing the scripts directory

* Rename template and address PR comments

* Add info about choosing a location and compatability

* Restart and run all

* Change new Markdown additions to be one sentence per line

* Add Style tips section

* Move location info to general advice, add datacube version cell

* Add instruction and example of including iPython magic commands

* Update to Template_notebook case style (#345)

* Add autoformatter details to template notebook and move tags info upward (#348)

* Add autoformatter details and move tags info upward

* Apply Black formatter to code cells

- changes from single quote to double quote

* Add initial set of scripts and funcs to `develop` (#349)

* Add initial set of scripts and funcs

* Use more explicit error handing in calculate_indices

* Make load_ard API more similar to dc.load

* Make product specification more explicit in load_ard

* Make parameters of calculate_indices more explicit

* Minor fix to documentation

* Rename Sentinel 2 collection for consistency/future proofing

As recommended by @sixy6e

* Fix incorrect band for NBR ratio

Good catch, @pkilgour!

* Simplify collection names which still following GA naming convention

Apparently 'ga_landsat_3' without including the word 'collection' is still valid according to the naming convention used in GA product names. This is simpler and easier to type, so let's go with that for now.

* Add Sentinel 2 NRT product names to documentation and exception handling

* Make normalisation step optional for calculate_indices (#351)

* Fix divide by 10000.0 in calculate_indices which became 1000.0

* Add optional normalise parameter to calculate_indices

* Add additional suggested urban indices to calculate_indices (#352)

* Add additional suggested urban indices to calculate_indices

Source: https://www.mdpi.com/2072-4292/11/3/345

* Add warning if user tries to run a coefficient index on unnormalised data

* Minor fix to documentation

* Add MSAVI and NDCI to calculate_indices (#353)

* Add MSAVI to calculate_indices

* Add NDCI Sentinel 2 index

* Sort indices by alphabetical order

* Change default normalisation to True

This makes the index calculations work correctly for coefficient bands. For back-compatible tasseled cap values, set `normalisation=False`

* Add Using_load_ard notebook to develop branch (#350)

* Add working version of load_ard notebook

* Update load_ard notebook

* Update load_ard notebook

* Upload version tested on Sandbox

* Update notebook for recent changes to load_ard

* Fix minor typos

* Fix AWEI water index formulas (#354)

* Add temporary resources for collection upgrade refresh (#355)

* Performance enhancements for load_ard (#358)

* Add placeholder dea_classificationtools script (#356)

* Add empty classification tools notebook

* Update text

* Adding temporary info to `develop` readme pages (#362)

* Add temporary info to readme files for each notebook directory

* Update README.rst

* Update README.rst

* Fix load_ard to preserve attributes when masking (#363)

* adding Exporting_geotiff notebook to Frequently_used_code (#371)

* first commit of exporting geotiffs notebook

* adding array_to_geotiff function to dea_datahandling

* updates to Export_geotiff notebook

* fix typos and improve some of the markdown"
"

* edited code to match pep8 standard

* edited markdown

* Minor formatting changes to Exporting geotiffs

* Add Working_with_time notebook (#366)

* Add Working_with_time notebook

* Add plt.show

* Add linear interpolation details to working with time

* Add after merging with remote changes

* adding array_to_geotiff function to Scripts/dea_datahandling (#370)

* adding array_to_geotiff function 

This simple function uses gdal to export arrays as geotiffs.  This is provided as an alternative to using datacube.helpers.write_geotiff which requires crs and geobox attributes to work.

* Minor change to formatting

* Add Integrating_external_data notebook (#373)

* Add Integrating_external_data notebook

* Minor updates to integrating external data

* Minor changes to text and plots after review

* Fix warning and add extra context to data load

* Fix load_ard dropping attributes when changing dtype (#381)

* Apply custom func along variables to preserve attrs when changing dtype

* Make new approach work on non-numeric variables

* Notebook s2 ndci (#367)

* Add initial set of scripts and funcs

* Use more explicit error handing in calculate_indices

* Make load_ard API more similar to dc.load

* Add template to form basis of Chlorophyll notebook

* Add necessary code for notebook

* Fix waterbody over time colourbar plot

- Add plots to show before and after NDCI and rgb

* Add necessary explanation text and clean code

* Change state to evaluated

- fix rgb to display images side by side

* Minor changes to text and outputs

- Carry over changes from existing version on master

- Remove instruction cells

- Change to smaller date range to lower load time

* Minor updates to chlorophyll notebook

* Minor updates to chlorophyll notebook

* Format new plots and fix tags

* Notebook coastalerosion (#391)

* Update coastal erosion notebook and add new funcs

* Commit correct notebook

* Add colorbar to contour plot and update reference

* Image segmentation notebook for Frequently_used_code (#390)

* first push of image seg notebook

* tidying image seg notebook

* minor edits to markdown

* reverse order of seg algorithms

* v minor markdown updates

* changed code to run using sentinel 2 data as C3 sample is deleted

* Style changes and small rearrange of Shepherd section

* Test on NCI and add open_rasterio as code

* Notebook changedetect (#392)

* Ignore Transitory_notebooks folder

* Modify template notebook to load and view data

* Add majority of analysis

- Includes experimental section on extracting ploygons. Not working currently as need to supply an affine transform to get the ploygons to display correctly.

* Add difference plots for significant change

- Change to using Sentinel-2 ARD

- Polygon not working, need affine transform

- Update tags

* Add explanation and instruction text

- Add section on exporting results to GeoTiff

* Change variable and associated text to improve clarity

* Minor updates for overhaul

* ran notebook on NCI and updated compatible environments comment

* commit with all cells run

* Fix #393 by not calculating aspect on the index dim (#394)

* Geomedian notebook for Frequently_used_code (#395)

* first commit of geomedian notebook

* cleaning up markdown

* Renamed notebook and minor changes

* Contour extraction notebook (#397)

* Add new contour extraction notebook

* Test on NCI and change wget to requests for compatability

* Add link to coastal erosion notebook and move markdown sentences to new lines

* Rasterizing vectors and vectorizing raster with rasterio (#399)

* first commit of rasterize & vectorie notebook

* vectorizing part done

* adding 'rasterize' code

* Minor formatting and code simplifications

* added ability to drop bands into the bandindices script (#400)

* added ability to drop bands into the bandindices script

* Minor formatting and spelling change

* improve dea_bandindices to calculate multiple indexes (#401)

* added changes to bandindices script to calculate multiple indexes

* Minor formatting update

* Add wofs_fuser to dea_datahandling (#403)

* Fix minor indent error (#405)

* Add calculating band indices notebook (#406)

* created notebook for calculating band indices

* fixed spaces and unused packages

* fixed tags

* ?

* Added example of multiple indices and some extra elaboration

* Virtual product example notebook (#409)

* Animated timeseries (#407)

* Add working version of animation code

* Add working version of animation code

* Test on NCI

* Add revised animated_timeseries notebook

* Add supplementary shapefile data

* Add requested changes

* Intertidal elevation notebook (#402)

* First pass of rewriting intertidal elevation notebook

* Add revised intertidal elevation notebook and fix for RGB

* Minor style updates and moved YAML file (#411)

* Notebook demonstrating cloud masking (#413)

* notebook demonstrating cloud masking

* distinction between clear and cloud-free

* Minor updates and move func to dea_datahandling

* Update to notebook template after refresh sprint (#412)

* Incremental update to notebook template

* Minor additions to General Advice cell

* Notebook wofs bitmask (#410)

* Create rough template for wofs bitmasking

* Update descriptions

- Include initial plot

- Output flag options neatly

* Add fractional cover masking example

Improve WOFL descriptions

* Remove unused cells

* Address review comments

 - use the wofs_fuser from scripts

 - update tech details to include correct products

 - small adjustments to text

* Add additional info and update tags

* Create draft notebook intro to jupyter (#376)

* Added notebook_Intro to Jupyter

* Updated background and introduction

* new sections on Jupyter cells

* first draft for review

* first draft for review

* updated to match beginners guide format

updated cell descriptions

* updated to match beginners guide format

updated cell descriptions

* Update text, add section on code cell status

* Add additional sections to Jupyter notebook and minor modifications

* Add open geotiff notebook to clean directory (#415)

* fix confusing typo in exporting geotiffs notebook (#417)

* Intro to products and measurements (#383)

* skeleton outline of notebook

* updated Loading Data section

* update the bulk contents of the notebook

* placeholder template notebook to intro ODC

* placeholder notebook for intro to measurements

* placeholder notebook for Intro to plotting

* placeholder notebook for Intro to products

* Clean branch and remove extra notebooks

* early draft of products listing

* early draft filtering products listing

* set max column width on dataframe

* complete the first full draft

editing still required

* add complete first draft

* Minor changes to text and apply condensed template format

* Incremental changes

* Run cells before committing

* Change PNG to JPG

* Remove product type for nicer table plotting

* Save widget state

* Remove Original_structure and add Practice_exercises dirs (#420)

* Add tidal phase calculation to tidal_tag function (#419)

* Add tidal phase calculation to tidal_tag function

This allows the user to estimate whether each satellite image was acquired during ebb or flow tides using `ebb_flow=True`. This information is added as a new variable in the dataset.

* Minor grammatical change

* Intro to loading data (#387)

* skeleton outline of notebook

* updated Loading Data section

* update the bulk contents of the notebook

* minor edits

* minor edits

* add structure to the introduction section of the notebook

* add detail to the customisation options for dc.load

* full draft

* complete full first draft

* Minor updates to text, changed variables from data to dataset to keep with existing convention

* Add query syntax section

* Clean loading data notebook

* Fix double formatting and rerun notebook

* Fix indented code, quotes, code var names and add local images

* Actually add local images

* Fix minor issues for DEA Docs

* Rename beginners notebooks, fix readthedocs issues, add style (#421)

* Rename notebooks, fix readthedocs issues, add style

* Fix minor typos in 01_Jupyter_notebooks.ipynb

* Fix minor typos in 03_Products_and_measurements.ipynb

* Fix minor typos in 04_Loading_data.ipynb

* Fix widget state for 03_Products_and_measurements.ipynb

* Fix bullet list in 04_Loading_data.ipynb

* Update style for coastal notebook and fix readthedocs errors (#422)

* Update style for coastal notebook and fix readthedocs errors

* Fix docs issues

* Intro to plotting (#384)

* skeleton outline of notebook

* updated Loading Data section

* update the bulk contents of the notebook

* placeholder template notebook to intro ODC

* placeholder notebook for intro to measurements

* placeholder notebook for Intro to plotting

* Clean branch and remove extra notebooks

* major edits to content

* complete first draft

notebook requires testing on NCI

* complete full first draft with revised intro structure

* complete full first draft with revised intro structure

* Add description of rgb function and three-colour imagery

* Added xarray plotting and dataviz sections to plotting notebook

* Remove accidently committed notebook

* One more attempt at committing the correct notebook...

* minor edits and re-wording around clipping colour stretches

* run all cells

* Minor style updates and fixes for readthedocs issues (#424)

* Minor style updates and fixes for readthedocs issues

* Add advice around Sphinx, fix minor issues

* S2 notebook (#425)

* first push of sentinel 2 notebook

* expanding on sentinel 2 loading options

* markdown edits

* fix typos

* Minor updates to text. Reduced lat,lon bounds to reduce load time

* Rename file to be consistent with naming convention

* Updates to S2 notebook (#426)

* Fix tags, add Sentinel-2 NRT

* Add additional detail about different between Definitive/NRT

* Change raw cell type to restructured text

* notebook now works on Sandbox, edited to reflect this (#423)

* notebook now works on Sandbox, edited to reflect this

* Additional style updates

* adding Dask-related UserWarning to load_ard (#428)

* adding dask UserWarning to load_ard

* Minor addition to doc-string and change to keep code at PEP8 line length

* Fix bug in Change Detection notebook (#429)

* Fix bug, minor markdown updates

* Update Sentinel 2 context and run all cells

* Intro to dea (#382)

* Added notebook_Intro to Jupyter

* Updated background and introduction

* new sections on Jupyter cells

* first draft for review

* first draft for review

* draft intro sections

* updated getting started section

* updated Data format section

* update the Getting Started section

* updated Band Designations section

* updated to-do list in Geolocation and Xarray sections

* update the To Continue section

* Clean branch and remove extra notebooks

* updated layout to match beginners guide format

* fleshed out content with material from docs.dea.ga.gov.au

needs completion of content, heavy editing for correctness

* save draft prior to major edit

* revisions to Introduction and Recommended cells

* update DEA sensors cell and add Figure 1

* updates to DEA datasets description

* add minor edits to DEA sensors cell

* revised surface reflectance cells

* update ARD sections

* update band naming cells

* drafted georeferencing section

* first draft of notebook

* renamed notebook

* Apply first round of content edits

- Apply template and markdown fixes to make consistent with other notebooks

- Fix hyperlinks to internal notebooks

Remove some of the overly technical information around Surface reflectance

* Test gif

* Add smaller res animation

* Add derived products

* Additional text updates

* Minor formatting update

* Add new beginner notebooks to index for readthedocs

* Apply minor gramatical fixes

* Make name of guide consistent in all notebooks

* Add largest_region function to dea_spatialtools (#432)

* Add largest_region function to dea_spatialtools

* Fix typo

* Rewrite contour code to be more efficient and user friendly (#435)

* Revise contour extraction notebook and function

* Update coastal erosion notebook and remove autoload from contour notebook

* Convert contour attribute data to length 10 string to stop shapefile writing from breaking

* Remove unary union from map_shapefile for 100x speedup

* Start updating intertidal notebook

* Rerun both coastal notebooks

* adding the inundation notebook to the real world examples folder (#369)

* adding the inundation notebook to the real world examples folder

* Update branch to include other Real world examples and supplementary folder

* adding the webscraping script needed for the inundation mapping tool

* updating the inundation notebook with a map and improved mark downs

* First edits to inundation notebook

* Latest updates to inundation mapping

* Add filter to valid stations and simplify figure

* Add file of valid stations

* updating with Caitlin's suggestions RE markdowns for the flow duration curve

* Add axis labels to plot, minor changes to text

* Classificationtools (#443)

* adding classification tools

* amended authorship

* amended date

* pep8

* fixed comment

* Added option in classification tools to use hdstats.geomedian (#444)

* Added option to use feature mean when extracting training data using a shapefile

* Updated to also calculate using geomedian.

* Generalised index creation process

* Minor updates to docstrings and code to align with PEP8 and Scipy documentation s

Reduced docstring line lengths to < 80 characters according to PEP8 (and some longer code lines)
Re-wrote some docstrings to use the Scipy/numpy docstring format

* Add Pan Sharpening Notebook (#442)

* Add Pan Sharpening Notebook

Add notebook that demonstrates pan sharpening Landsat 8

* Updates to pansharpening notebook

* Move context upward

* Rerun all cells

* Update of all notebooks to fix outstanding DEA Docs issues (#446)

* Add TOC to README files

* Fix styling issues that occur for DEA docs

* Fix issues with list formatting for DEA docs

* Test to check heading format for DEA docs

* Add recently made notebooks to TOCs, for docs display

* Minor changes to text to make instructions more direct

* Make minor changes to recommended steps

* Fix indentation issues for figure caoptions.

* Make minor fix in next steps

* Minor changes to text to make instructions clearer

* Rerun the DCViewer Utility

* Fix NCI tag

* Simplify language throughout

* Simplify language throughout

* Add reference to existing Beginner's guide notebook

* Rerun notebook, update date modified

* Update notebook to match template style

* Update date modified

* Update to reflect now working on Sandbox

* Update WOfS bitmasking to match template.

* Update Calculating band indicees to match template

* Update plot display method

* Fix plot display issue in WOfS bitmasking and Calculating band indices

* Rerun all cells in order for calculating band indices

* Check different plotting call for sphinx format

* Fix hyperlink in Contour extraction

* Update Exporting geotiffs to match template

* Fix vertices typo in Contour extraction

* Update Geomedian composites to match template

* Update Image segmentation to match template

* Update Integrating external data to match template

* Update tags in Integrating external data

* Update tags in Exporting geotiffs

* Update Masking data to mo match template

* Update tags in Masking data

* Update Open Geotiff to match template

* Update Rasterize vectorize to match template

* Update Using load ard to match template

* Remove technical details cell from Using load ard

* Update Virtual products to match template

* Update Working with time to match template

* Update Chlorophyll monitoring to match template

* Add getting started section to Chlorophyll monitoring

* Remove unecessary cell from Chlorophyll monitoring

* Update Coastal erosion to match template

* Fix widget state for Coastal errosion

* Update Intertidal elevation to match template

* Incremental updates on Sandbox

* Test first four notebooks on NCI

* Reduce length of external tide file to save file size

* Test geomedian and segmentation notebooks on NCI

* Finish testing all frequently used, update date last run

* Test all real world notebooks on NCI

* Test all beginners notebooks on NCI

* Fix timestep plotted in intertidal dem notebook

* Test Sentinel 2 notebook on NCI

* Updates to load GeoTIFF and NetCDF notebook and relevant func

* Remove non-used imports

* Fix formatting and update markdown text

* Add revised exporting NetCDF notebook

* Update exporting geotiff notebook to match NetCDF notebook

* Fix tags in beginners notebooks

* Clarify S2 NRT is not available on NCI

* Fix non-run S2 notebook

* Fix broken reference and add new notebooks to README

* Updated dea-notebooks readme (#450)

* Updated dea-notebooks readme

* Add broad overview

* Updates to branches section

* Update template and using git

* Attempt at fixing indented list formatting

* Fix indented list formatting

* Add PR section and move template down

* Add contributing to master section

* Update wiki urls

* Add draft PR section and links to other repos

* Phrasing

* Add wide logo for readme

* Add logo image

* Update directory readme

* Update DEA dataset directory rst

* Capitalisation

* Update frequently used code directory rst

* Update real world examples directory rst

* Add "Show image on a web map" example notebook (#447)

* Add "Show image on a web map" example notebook

* Remove footer cell, fix a comment

* Formatting updates and add tags

* Update compatability and widget status

* Don't user show_datasets instead create Map manually

- demonstrate ipyleaflet.Map creation
- demonstrate adding GeoJSON object to the map

* Correct doc: it's not NRT

* Minor markdown update

* Add new Scientific_workflows directory to DEA Notebooks (#451)

* Move practice exercises into real world examples

* Add new directory for scientific workflows

* Add scientific workflows description to repo readme

* Add link to wiki getting started guides to readme (#453)

* Add missing Imagery_on_web_map.ipynb notebook to index (#454)

* DEA_datasets notebook for new Tsmask (#455)

* Add new TSmask notebook

* Minor formatting fix for S2 notebook

* Update rst index

* Fix grammar and code comment

* Quick review of the surface reflectance content (#457)

* Various changes; NBAR-T to NBART (the CMI product docs don't hyphenise NBART); Corrected a sentence saying that surface reflectance enables consistent comparison (this is not true as different sensor are typically measuring different portions of the spectrum, it is however true for within a sensor); Corrected a sentence regarding UTM being alinged to WGS84 (it is not aligned but uses the WGS84 ellipsoid to model the UTM projection); the DSM is not the same resolution as the image being processed but resampled to the same resolution; the NBART nodata value is -999 not -9999

* Minor update to text

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Add temporary header to readme pointing to master_archive branch (#458)

* Water classification notebook (#459)

* Overhaul of original water classification notebook to meet new standards

* Add new notebook to README

* Updates to water detection notebook

* Restart rerun

* Final run on Sandbox

* Fix missing restructured text tags format

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Crop health notebook (#461)

* Add Agriculture notebook as Crop health. Update to work with overhauled functions

* Add function to transform coords of drawn polygons, required for crophealth app

* Reformatted notebook to match new style

* Change data to ARD and run analysis for a year, with saved outputs

* Add minor changes to crop health notebook

* Fix tags format

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Update dea_datahandling.py (#463)

Make clear that Sentinel 2 NRT not available via VDI.

* Mining rehabilitation notebook overhaul (#464)

* Overhaul of mining notebook and associated functions

* Removing Crop_health.ipynb from README to stop it being built by DEA-Docs

* Minor updates to mining notebook

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Shipping lanes notebook overhaul (#465)

* Add overhauled shipping lanes notebook

* Minor updates to shipping lanes notebook;

* Suggested fixes

* Fix formatting

* Change program to mission

Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* Pre-merge patch for develop (#467)

* Minor fix to line in .gitignore

* Adding text to README for github

* Remove unused folders from Supplementary data TOC

Co-authored-by: Caitlin Adams <[email protected]>
Co-authored-by: cbur24 <[email protected]>
Co-authored-by: Bex Dunn <[email protected]>
Co-authored-by: Imam Alam <[email protected]>
Co-authored-by: Belle <[email protected]>
Co-authored-by: erialC-P <[email protected]>
Co-authored-by: Cate Kooymans <[email protected]>
Co-authored-by: Katey <[email protected]>
Co-authored-by: S Chua <[email protected]>
Co-authored-by: David Gavin <[email protected]>
Co-authored-by: Kirill Kouzoubov <[email protected]>
Co-authored-by: Josh Sixsmith <[email protected]>
Co-authored-by: Claire <[email protected]>