Merge pull request #63 from sat-utils/develop

publish 0.2.0

.circleci/config.yml

@@ -44,6 +44,7 @@ jobs:
             pwd
             pip install -r requirements.txt
             pip install -r requirements-dev.txt
+            export SATUTILS_API_URL=https://n34f767n91.execute-api.us-east-1.amazonaws.com/prod
             pytest --cov satsearch test/
       - save_cache:
           key: v1-dependencies-{{ checksum "requirements.txt"}}

CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+
+## [v0.2.0] - 2019-01-31
+
+## Changed
+- Works with version 0.2.0 of sat-api (STAC 0.6.x)
+- Major refactor, uses sat-stac library
+
+
+## [v0.1.0] - 2018-10-25
+
+Initial Release
+
+[Unreleased]: https://github.com/sat-utils/sat-search/compare/master...develop
+[v0.2.0]: https://github.com/sat-utils/sat-search/compare/0.1.0...v0.2.0
+[v0.1.0]: https://github.com/sat-utils/sat-search/tree/0.1.0

README.md

@@ -1,72 +1,80 @@
-# SAT-SEARCH
+# sat-search
 
 [![CircleCI](https://circleci.com/gh/sat-utils/sat-search.svg?style=svg&circle-token=a66861b5cbba7acd4abd7975f804ab061a365e1b)](https://circleci.com/gh/sat-utils/sat-search)
 
-Sat-search is a Python 2/3 library and a command line tool for discovering and downloading publicly available satellite imagery using a conformant API such as [sat-api](https://github.com/sat-utils/sat-api).
+Sat-search is a Python 3 library and a command line tool for discovering and downloading publicly available satellite imagery using a conformant API such as [sat-api](https://github.com/sat-utils/sat-api).
 
-The STAC version supported by a given version of sat-api is shown in the table below. Additional information can be found in the [CHANGELOG](CHANGELOG.md)
 
-| sat-search | STAC  |
-| ---------- | ----  |
-| 0.1.0      | 0.5.0 |
-| 0.2.0      | 0.6.0 |
+## Installation
 
+Sat-search is a very lightweight application, with the only dependency being [sat-stac](https://github.com/sat-utils/sat-stac), which in turn has two dependencies: `requests` and `python-dateutil`. To install sat-search from PyPi:
 
-## Installation
-It is recommended to use [pyenv](https://github.com/pyenv/pyenv) and [virtualenv](https://virtualenv.pypa.io/en/latest/) to to control Python versions and installed dependencies. sat-search can be conveniently installed from PyPi:
+```bash
+$ pip install sat-search
+```
+
+From source repository:
 
-    # install the latest release version
-    $ pip install sat-search
+```bash
+$ git clone https://github.com/sat-utils/sat-search.git
+$ cd sat-search
+$ pip install .
+```
+
+#### Versions
+The latest version of sat-search is 0.2.0, which uses the STAC spec v0.6.0. To install other versions of sat-search, specify the version in the call to pip. 
+
+```bash
+pip install sat-search==0.2.0
+```
+
+The table below shows the corresponding versions between sat-search and STAC. Additional information can be found in the [CHANGELOG](CHANGELOG.md)
+
+| sat-search | STAC  |
+| --------   | ----  |
+| 0.1.x      | 0.5.0 |
+| 0.2.x      | 0.6.0 |
 
-Sat-search is a very lightweight application, with the only dependency being [sat-stac](https://github.com/sat-utils/sat-stac), which in turn has two dependencies: `requests` and `python-dateutil`.
 
 ## Using sat-search
 
-With sat-search you can search a STAC compliant API with full querying support (if supported by the API). Search results are saved as a GeoJSON FeatureCollection and can be loaded later. Assets can be easily downloaded by the key, or color if provided.
+With sat-search you can search a STAC compliant API with full querying support (if supported by the API). Search results can be saved as a GeoJSON file then loaded later. Assets can be downloaded by the asset key, or "color" (common_name of the band) if provided.
+
+Sat-search is a Python 3 library that can incorporated into other applications. A [Jupyter notebook tutorial](tutorial-1.ipynb) is included that covers all the main features of the library.
 
-Sat-search comes with a Command Line Interface (CLI), but is also a Python library that can be incorporated into other applications. This README only covers use of the CLI.
+Sat-search also comes with a Command Line Interface (CLI), which is exaplained more below.
 
 #### The CLI
 The sat-search CLI has an extensive online help that can be printed with the `-h` switch.
+
 ```
 $ sat-search -h
 usage: sat-search [-h] {search,load} ...
 
-sat-search (v0.2.0b1)
+sat-search (v0.2.0)
 
 positional arguments:
   {search,load}
-    search       Perform new search of scenes
-    load         Load scenes from previous search
+    search       Perform new search of items
+    load         Load items from previous search
 
 optional arguments:
   -h, --help     show this help message and exit
 ```
 
-As can be seen there are two subcommands, each of which has it's own online help (i.e. "sat-search search -h" and "sat-search load -h") and will be discussed in detail below.
+As can be seen there are two subcommands, `search` and `load`, each of which has it's own help.
 
-#### Searching
+#### `search`
 
 ```
-usage: sat-search [-h] {search,load} ...
-
-sat-search (v0.2.0b1)
-
-positional arguments:
-  {search,load}
-    search       Perform new search of items
-    load         Load items from previous search
-
-optional arguments:
-  -h, --help     show this help message and exit
 (satutils) mhanson@clavius:~/devseed/sat-utils/sat-search/scratch$ sat-search search -h
 usage: sat-search search [-h] [--version] [-v VERBOSITY]
-                         [--print_md [PRINT_MD [PRINT_MD ...]]] [--print_cal]
-                         [--save SAVE] [-c COLLECTION]
+                         [--print-md [PRINTMD [PRINTMD ...]]] [--print-cal]
+                         [--save SAVE] [-c COLLECTION] [--ids [IDS [IDS ...]]]
                          [--bbox BBOX BBOX BBOX BBOX]
                          [--intersects INTERSECTS] [--datetime DATETIME]
-                         [--sort SORT] [-p [PROPERTY [PROPERTY ...]]]
-                         [--url URL]
+                         [--sort [SORT [SORT ...]]] [--found]
+                         [-p [PROPERTY [PROPERTY ...]]] [--url URL]
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -76,51 +84,64 @@ optional arguments:
                         2)
 
 output options:
-  --print_md [PRINT_MD [PRINT_MD ...]]
+  --print-md [PRINTMD [PRINTMD ...]]
                         Print specified metadata for matched scenes (default:
                         None)
-  --print_cal           Print calendar showing dates (default: False)
+  --print-cal           Print calendar showing dates (default: False)
   --save SAVE           Save results as GeoJSON (default: None)
 
 search options:
   -c COLLECTION, --collection COLLECTION
                         Name of collection (default: None)
+  --ids [IDS [IDS ...]]
+                        One or more scene IDs from provided collection
+                        (ignores other parameters) (default: None)
   --bbox BBOX BBOX BBOX BBOX
                         Bounding box (min lon, min lat, max lon, max lat)
                         (default: None)
   --intersects INTERSECTS
                         GeoJSON Feature (file or string) (default: None)
   --datetime DATETIME   Single date/time or begin and end date/time (e.g.,
                         2017-01-01/2017-02-15) (default: None)
-  --sort SORT           Sort by fields (default: None)
+  --sort [SORT [SORT ...]]
+                        Sort by fields (default: None)
+  --found               Only output how many Items found (default: False)
   -p [PROPERTY [PROPERTY ...]], --property [PROPERTY [PROPERTY ...]]
                         Properties of form KEY=VALUE (<, >, <=, >=, =
                         supported) (default: None)
-  --url URL             URL of the API (default: https://sat-api-
-                        dev.developmentseed.org)
+  --url URL             URL of the API (default: https://n34f767n91.execute-
+                        api.us-east-1.amazonaws.com/prod)
 ```
 
 **Search options**
 
+- **collection** - Search only a specific collection. This is a shortcut, collection can also be provided as a property (e.g., `-p "collection=landsat-8-l1")
+- **ids** - Fetch the Item for the provided IDs in the given collection (collection must be provided). All other search options will be ignored.
 - **intersects** - Provide a GeoJSON Feature string or the name of a GeoJSON file containing a single Feature that is a Polygon of an AOI to be searched.
 - **datetime** - Provide a single partial or full datetime (e.g., 2017, 2017-10, 2017-10-11, 2017-10-11T12:00), or two seperated by a slash that defines a range. e.g., 2017-01-01/2017-06-30 will search for scenes acquired in the first 6 months of 2017.
-- **property** - Allows searching for any other scene properties by providing the pair as KEY=VALUE (e.g. `-p "landsat:row=42"`, `-p "eo:cloud_cover<10"`)
+- **property** - Allows searching for any other scene properties by providing the pair as a string (e.g. `-p "landsat:row=42"`, `-p "eo:cloud_cover<10"`). Supported symbols include: =, <, >, >=, and <=
+- **sort** - Sort by specific properties in ascending or descending order. A list of properties can be provided which will be used for sorting in that order of preference. By default a property will be sorted in descending order. To specify the order the property can be preceded with '<' (ascending) or '>' (descending). e.g., `--sort ">datetime" "<eo:cloud_cover" will sort by descending date, then by ascending cloud cover
+- **found** - This will print out the total number of scenes found, then exit without fetching the actual items. 
 - **url** - The URL endpoint of a STAC compliant API, this can also be set with the environment variable SATUTILS_API_URL
 
 **Output options**
 These options control what to do with the search results, multiple switches can be provided.
 
-- **print_md** - Prints a list of specific metadata fields for all the scenes. If given without any arguments it will print a list of the dates and scene IDs. Otherwise it will print a list of fields that are provided. (e.g., --print_md date eo:cloud_cover eo:platform will print a list of date, cloud cover, and the satellite platform such as WORLDVIEW03)
-- **print_cal** - Prints a text calendar with specific days colored depending on the platform of the scene (e.g. landsat-8), along with a legend.
+- **print-md** - Prints a list of specific metadata fields for all the scenes. If given without any arguments it will print a list of the dates and scene IDs. Otherwise it will print a list of fields that are provided. (e.g., --print-md date eo:cloud_cover eo:platform will print a list of date, cloud cover, and the satellite platform such as WORLDVIEW03)
+- **print-cal** - Prints a text calendar (see iumage below) with specific days colored depending on the platform of the scene (e.g. landsat-8), along with a legend.
 - **save** - Saves results as a FeatureCollection. The FeatureCollection 'properties' contains all of the arguments used in the search and the 'features' contain all of the individual scenes, with individual scene metadata merged with collection level metadata (metadata fields that are the same across all one collection, such as eo:platform)
 
-#### Loading
+![](images/calendar.png)
+
+
+#### `load`
+
 Scenes that were previously saved with `sat-search search --save ...` can be loaded with the `load` subcommand.
 
 ```
 $ sat-search load -h
 usage: sat-search load [-h] [--version] [-v VERBOSITY]
-                       [--print_md [PRINT_MD [PRINT_MD ...]]] [--print_cal]
+                       [--print-md [PRINTMD [PRINTMD ...]]] [--print-cal]
                        [--save SAVE] [--datadir DATADIR] [--filename FILENAME]
                        [--download [DOWNLOAD [DOWNLOAD ...]]]
                        items
@@ -136,10 +157,10 @@ optional arguments:
                         2)
 
 output options:
-  --print_md [PRINT_MD [PRINT_MD ...]]
+  --print-md [PRINTMD [PRINTMD ...]]
                         Print specified metadata for matched scenes (default:
                         None)
-  --print_cal           Print calendar showing dates (default: False)
+  --print-cal           Print calendar showing dates (default: False)
   --save SAVE           Save results as GeoJSON (default: None)
 
 download options:
@@ -151,7 +172,7 @@ download options:
                         Download assets (default: None)
 ```
 
-Note that while the search options are gone, output options are still available and can be used with the search results loaded from the file. There is also a new series of options now, for downloading data.
+Note that while the search options are gone, output options are still available and can be used with the search results loaded from the file. There is also a new series of options for downloading data.
 
 #### Downloading assets
 When loading results from a file, the user now has the option to download assets from the scenes.
@@ -163,14 +184,28 @@ These control the downloading of assets. Both datadir and filename can include m
 - **download** - Provide a list of keys to download these assets. More information on downloading data is provided below.
 
 **Metadata patterns**
-Metadata patterns can be within **datadir** and **filename** in order to have custom path and filenames based on the scene metadata. For instance specifying datadir as "./${eo:platform}/${date}" will save assets for each scene under directories of the platform and the date. So a landsat-8 scene from June 20, 2018 will have it's assets saved in a directory './landsat-8/2017-06-20'. For filenames these work exactly the same way, except the appropriate extension will be used at the end of the filename, depending on the asset.
+Metadata patterns can be used in **datadir** and **filename** in order to have custom path and filenames based on the Item metadata. For instance specifying datadir as "./${eo:platform}/${date}" will save assets for each Item under directories of the platform and the date. So a landsat-8 Item from June 20, 2018 will have it's assets saved in a directory './landsat-8/2017-06-20'. For filenames these work exactly the same way, except the filename will contain a suffix containing the asset key and the appropriate extension.
+
+```
+    sat-search load scenes.json --download thumbnail MTL
+```
+
+In this case the defaults for `datadir` ("./${eo:platform}/${date}") and `filename` ("${id}") are used so the download files are saved like this:
+
+```
+landsat-8/
+└── 2018-10-02
+    ├── LC80090292018275LGN00_MTL.txt
+    ├── LC80090292018275LGN00_thumbnail.jpg
+    ├── LC80090302018275LGN00_MTL.txt
+    └── LC80090302018275LGN00_thumbnail.jpg
+```
+
+A shortcut to download all of the assets is available by providing "ALL" as the key to download. This will download every asset for every item.
 
-**Assets**
-The thumbnail for each scene in a *scenes.json* file can be downloaded with
 ```
-    sat-search load scenes.json --download assetname1 assetname2 ...
+   sat-search load scenes.json --download ALL
 ```
-The thumbnails will be saved using a directory and filename according to the `datadir` and `filename` options, and will also have a '_thumbnail` suffix. When thumbnails are downloaded an ESRI Worldfile (.wld) file is created, which is a sidecar file that describes the coordinates and resolution of the images. This enables the thumbnails to be viewed in a GIS program like QGIS in their proper geographical location. The world file does not set the spatial reference system used (lat/lon, or WGS-84, or EPSG:4326), so when opened in QGIS it will need to be selected (EPSG:4326).
 
 ## About
 sat-search was created by [Development Seed](<http://developmentseed.org>) and is part of a collection of tools called [sat-utils](https://github.com/sat-utils).

requirements.txt

@@ -1 +1 @@
-sat-stac~=0.1.0rc5
+sat-stac~=0.1.1

satsearch/main.py

@@ -6,52 +6,63 @@
 from satstac import Items
 from satsearch.parser import SatUtilsParser
 
+import satsearch.config as config
 
-def main(items=None, print_md=None, print_cal=False,
+
+def main(items=None, printmd=None, printcal=False, found=False,
          save=None, download=None, **kwargs):
     """ Main function for performing a search """
+
     if items is None:
-        # get items from search
-        search = Search(**kwargs)
+        ## if there are no items then perform a search
+        search = Search.search(**kwargs)
+        if found:
+            num = search.found()
+            print('%s items found' % num)
+            return num
         items = search.items()
     else:
+        # otherwise, load a search from a file
         items = Items.load(items)
 
+    print('%s items found' % len(items))
+
     # print metadata
-    if print_md is not None:
-        items.print_summary(print_md)
+    if printmd is not None:
+        print(items.summary(printmd))
 
     # print calendar
-    if print_cal:
-        print(items.text_calendar())
+    if printcal:
+        print(items.calendar())
 
     # save all metadata in JSON file
     if save is not None:
         items.save(filename=save)
 
-    print('%s items found' % len(items))
-
-    # download files given keys
+    # download files given `download` keys
     if download is not None:
+        if 'ALL' in download:
+            # get complete set of assets
+            download = set([k for i in items for k in i.assets])
         for key in download:
-            items.download(key=key)
+            items.download(key=key, path=config.DATADIR, filename=config.FILENAME)
 
     return items
 
 
 def cli():
     parser = SatUtilsParser.newbie(description='sat-search (v%s)' % __version__)
-    args = parser.parse_args(sys.argv[1:])
+    kwargs = parser.parse_args(sys.argv[1:])
 
-    # read the GeoJSON file
-    if 'intersects' in args:
-        if os.path.exists(args['intersects']):
-            with open(args['intersects']) as f:
-                args['intersects'] = json.dumps(json.loads(f.read()))
+    # if a filename, read the GeoJSON file
+    if 'intersects' in kwargs:
+        if os.path.exists(kwargs['intersects']):
+            with open(kwargs['intersects']) as f:
+                kwargs['intersects'] = json.loads(f.read())
 
-    cmd = args.pop('command', None)
+    cmd = kwargs.pop('command', None)
     if cmd is not None:
-        return main(**args)
+        main(**kwargs)
 
 
 if __name__ == "__main__":