distil-ge-fetch

Main Scripts

fetch_data.py

Python script for fetching Google Earth Engine tiles based on an area of interest, geohash level, and start / end time. Can also fetch based on an input JSON file. Note that an activated Google Earth Engine account is required.

Example usage for fetch:

python3 fetch_data.py \
    --coverage_file data/shire.geojson \
    --poi_file data/shire_points.geojson \
    --outdir data/shire_test \
    --start_date 2018-01-01 \
    --end_date 2019-02-01 \
    --precision 5 \
    --interval 30 \
    --n_jobs 4 \
    --sampling 1.0 \
    --save_requests

The call above fetches tiles within the polygon defined by shire.geojson, between 2018/01/01 and 2019/02/01 that fall within 30 day intervals. The tiles are defined at geohash level 5, and no subsampling is performed. In addition, tiles that geographically and temporally intersect the points of interest defined in shire_points.geojson are fetched. The end result is a directory containing area and poi directories contained zipped tile files, named by geoash and date. Example:

data/shire_test/poi/scu6k_2019-12-30.zip

create_dataset.py

A script is also included to unzip the fetched archives and name them according to the sentinel-2 standard. They can be unzipped into a flat file structure, or folders can be created based on supplied label names, where the poi data from the previous step is the positive_label, and the area is the negative_label. Example:

python3 create_dataset.py \
    --download_location data/shire_test \
    --output_location data/shire_test_sentinel \
    --positive_label swarm_sighting \
    --negative_label no_swarm_sighting

The output of this step is of a format suitable for import directly into Distil, and should be easy to adapt to other modelling workflows.

Additional Scripts

csv_to_geojson.py

Converts a CSV file with columns that have a date/time, latitude, and longitude column into a geojson file point set, with the date applied as metadata to each point. Example:

python3 csv_to_geojson.py \
    --features_csv data/Hoppers.csv \
    --longitude_col X \
    --latitude_col Y \
    --date_col STARTDATE \
    --output_file data/hoppers.geojson

The resulting CSV file is formatted for use as POI inputs with the featch_dataset.py script.

hysplit_extract.py

Takes a directory containing projected swarm flight paths generated by the HYSPLIT locust forecast tool and uses it to generate a CSV file consisting of columns for the path vertices, altitude and time. The CSV file can be directly imported into Distil or adapted to other modelling workflows. Example:

python3 hysplit_extract.py \
    --input_dir data/swarm_7510 \
    --output_file data/remote_sensing/swarm_7510.csv

optram.py

The optram script will calculate variables necessary for calculating soil moisture for a given geohash. optram script parameters:

• download_location
  • Description: a path pointing to output of the fetch_data.py script
  • Required
• precision
  • Description: The number of precision for the spatial hash for example given geohash 347ed a precision of 3 will put the hash in bucket 347
  • Not Required
  • Defaults to 3
  • Note: The more precise the less resources required
• ouput_location
  • Description: a path pointing to a folder to output the json to
  • Not Required
  • Defaults to working directory

example:

python3 optram.py --download_location data/shire --precision 4 \--output_location data