Merge pull request #61 from sat-utils/develop

release 0.2.0b1

README.md

@@ -4,33 +4,35 @@
 
 Sat-search is a Python 2/3 library and a command line tool for discovering and downloading publicly available satellite imagery using a conformant API such as [sat-api](https://github.com/sat-utils/sat-api).
 
-The legacy version of sat-search (<1.0.0) can be used with the legacy version of sat-api (<1.0.0), currently deployed at https://api.developmentseed.org/satellites.
+The STAC version supported by a given version of sat-api is shown in the table below. Additional information can be found in the [CHANGELOG](CHANGELOG.md)
+
+| sat-search | STAC  |
+| ---------- | ----  |
+| 0.1.0      | 0.5.0 |
+| 0.2.0      | 0.6.0 |
+
 
 ## Installation
 It is recommended to use [pyenv](https://github.com/pyenv/pyenv) and [virtualenv](https://virtualenv.pypa.io/en/latest/) to to control Python versions and installed dependencies. sat-search can be conveniently installed from PyPi:
 
     # install the latest release version
     $ pip install sat-search
 
-Sat-search is a very lightweight application, with the only dependency being requests.
+Sat-search is a very lightweight application, with the only dependency being [sat-stac](https://github.com/sat-utils/sat-stac), which in turn has two dependencies: `requests` and `python-dateutil`.
 
 ## Using sat-search
 
-Sat-search has several features:
+With sat-search you can search a STAC compliant API with full querying support (if supported by the API). Search results are saved as a GeoJSON FeatureCollection and can be loaded later. Assets can be easily downloaded by the key, or color if provided.
 
-- search catalog
-- STAC compliant interface
-- save results of a search
-- load results of a search
-- download assets (e.g. thumbnails, data files) of the results
+Sat-search comes with a Command Line Interface (CLI), but is also a Python library that can be incorporated into other applications. This README only covers use of the CLI.
 
 #### The CLI
 The sat-search CLI has an extensive online help that can be printed with the `-h` switch.
 ```
 $ sat-search -h
 usage: sat-search [-h] {search,load} ...
 
-sat-search (v1.0.0b8)
+sat-search (v0.2.0b1)
 
 positional arguments:
   {search,load}
@@ -46,13 +48,25 @@ As can be seen there are two subcommands, each of which has it's own online help
 #### Searching
 
 ```
-$ sat-search search -h
+usage: sat-search [-h] {search,load} ...
+
+sat-search (v0.2.0b1)
+
+positional arguments:
+  {search,load}
+    search       Perform new search of items
+    load         Load items from previous search
+
+optional arguments:
+  -h, --help     show this help message and exit
+(satutils) mhanson@clavius:~/devseed/sat-utils/sat-search/scratch$ sat-search search -h
 usage: sat-search search [-h] [--version] [-v VERBOSITY]
                          [--print_md [PRINT_MD [PRINT_MD ...]]] [--print_cal]
-                         [--save SAVE] [--append] [-c [C:ID [C:ID ...]]]
+                         [--save SAVE] [-c COLLECTION]
+                         [--bbox BBOX BBOX BBOX BBOX]
                          [--intersects INTERSECTS] [--datetime DATETIME]
-                         [--eo:cloud_cover EO:CLOUD_COVER]
-                         [-p [PARAM [PARAM ...]]] [--url URL]
+                         [--sort SORT] [-p [PROPERTY [PROPERTY ...]]]
+                         [--url URL]
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -67,33 +81,30 @@ output options:
                         None)
   --print_cal           Print calendar showing dates (default: False)
   --save SAVE           Save results as GeoJSON (default: None)
-  --append              Append scenes to GeoJSON file (specified by save)
-                        (default: False)
 
 search options:
-  -c [C:ID [C:ID ...]], --c:id [C:ID [C:ID ...]]
-                        Name(s) of collection (default: None)
+  -c COLLECTION, --collection COLLECTION
+                        Name of collection (default: None)
+  --bbox BBOX BBOX BBOX BBOX
+                        Bounding box (min lon, min lat, max lon, max lat)
+                        (default: None)
   --intersects INTERSECTS
                         GeoJSON Feature (file or string) (default: None)
   --datetime DATETIME   Single date/time or begin and end date/time (e.g.,
-                        2017-01-01/2017-02-15 (default: None)
-  --eo:cloud_cover EO:CLOUD_COVER
-                        Range of acceptable cloud cover (e.g., 0/20) (default:
-                        None)
-  -p [PARAM [PARAM ...]], --param [PARAM [PARAM ...]]
-                        Additional parameters of form KEY=VALUE (default:
-                        None)
-  --url URL             URL of the API (default: https://sat-
-                        api.developmentseed.org)
+                        2017-01-01/2017-02-15) (default: None)
+  --sort SORT           Sort by fields (default: None)
+  -p [PROPERTY [PROPERTY ...]], --property [PROPERTY [PROPERTY ...]]
+                        Properties of form KEY=VALUE (<, >, <=, >=, =
+                        supported) (default: None)
+  --url URL             URL of the API (default: https://sat-api-
+                        dev.developmentseed.org)
 ```
 
 **Search options**
 
-- **c:id** - A list of names of collections (i.e. sensors). The collections supported depend on the API, and for sat-api can be seen at the [collections endpoint](https://sat-api.developmentseed.org/collections). If one or more collections are not defined, all collections are searched.
 - **intersects** - Provide a GeoJSON Feature string or the name of a GeoJSON file containing a single Feature that is a Polygon of an AOI to be searched.
 - **datetime** - Provide a single partial or full datetime (e.g., 2017, 2017-10, 2017-10-11, 2017-10-11T12:00), or two seperated by a slash that defines a range. e.g., 2017-01-01/2017-06-30 will search for scenes acquired in the first 6 months of 2017.
-- **eo:cloud_cover** - Provide a single percent cloud cover to match (e.g., 0) or two numbers separated by a slash indicating the range of acceptable cloud cover (e.g., 0/20 searches for scenes with 0% - 20% cloud cover).
-- **param** - Allows searching for any other scene properties by providing the pair as KEY=VALUE (e.g. `-p landsat:row=42`)
+- **property** - Allows searching for any other scene properties by providing the pair as KEY=VALUE (e.g. `-p "landsat:row=42"`, `-p "eo:cloud_cover<10"`)
 - **url** - The URL endpoint of a STAC compliant API, this can also be set with the environment variable SATUTILS_API_URL
 
 **Output options**
@@ -102,7 +113,6 @@ These options control what to do with the search results, multiple switches can
 - **print_md** - Prints a list of specific metadata fields for all the scenes. If given without any arguments it will print a list of the dates and scene IDs. Otherwise it will print a list of fields that are provided. (e.g., --print_md date eo:cloud_cover eo:platform will print a list of date, cloud cover, and the satellite platform such as WORLDVIEW03)
 - **print_cal** - Prints a text calendar with specific days colored depending on the platform of the scene (e.g. landsat-8), along with a legend.
 - **save** - Saves results as a FeatureCollection. The FeatureCollection 'properties' contains all of the arguments used in the search and the 'features' contain all of the individual scenes, with individual scene metadata merged with collection level metadata (metadata fields that are the same across all one collection, such as eo:platform)
-- **append** - The save option will always create a new file, even overwriting an existing one. If *append* is provided then the scenes will be appended to the FeatureCollection given by the save filename.
 
 #### Loading
 Scenes that were previously saved with `sat-search search --save ...` can be loaded with the `load` subcommand.
@@ -111,13 +121,12 @@ Scenes that were previously saved with `sat-search search --save ...` can be loa
 $ sat-search load -h
 usage: sat-search load [-h] [--version] [-v VERBOSITY]
                        [--print_md [PRINT_MD [PRINT_MD ...]]] [--print_cal]
-                       [--save SAVE] [--append] [--datadir DATADIR]
-                       [--filename FILENAME]
+                       [--save SAVE] [--datadir DATADIR] [--filename FILENAME]
                        [--download [DOWNLOAD [DOWNLOAD ...]]]
-                       scenes
+                       items
 
 positional arguments:
-  scenes                GeoJSON file of scenes
+  items                 GeoJSON file of Items
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -132,8 +141,6 @@ output options:
                         None)
   --print_cal           Print calendar showing dates (default: False)
   --save SAVE           Save results as GeoJSON (default: None)
-  --append              Append scenes to GeoJSON file (specified by save)
-                        (default: False)
 
 download options:
   --datadir DATADIR     Directory pattern to save assets (default:
@@ -153,10 +160,10 @@ When loading results from a file, the user now has the option to download assets
 These control the downloading of assets. Both datadir and filename can include metadata patterns that will be substituted per scene.
 - **datadir** - This specifies where downloaded assets will be saved to. It can also be specified by setting the environment variable SATUTILS_DATADIR.
 - **filename** - The name of the file to save. It can also be set by setting the environment variable SATUTILS_FILENAME
-- **download** - Provide a list of keys to download these assets. For DG currently only **thumbnail** and **full** are supported. More information on downloading data is provided below.
+- **download** - Provide a list of keys to download these assets. More information on downloading data is provided below.
 
 **Metadata patterns**
-Metadata patterns can be within **datadir** and **filename** in order to have custom path and filenames based on the scene metadata. For instance specifying datadir as "./${eo:platform}/${date}" will save assets for each scene under directories of the platform and the date. So a WorldView-3 scene from June 20, 2018 will have it's assets saved in a directory './WORLDVIEW03/2017-06-20'. For filenames these work exactly the same way, except the appropriate extension will be used at the end of the filename, depending on the asset.
+Metadata patterns can be within **datadir** and **filename** in order to have custom path and filenames based on the scene metadata. For instance specifying datadir as "./${eo:platform}/${date}" will save assets for each scene under directories of the platform and the date. So a landsat-8 scene from June 20, 2018 will have it's assets saved in a directory './landsat-8/2017-06-20'. For filenames these work exactly the same way, except the appropriate extension will be used at the end of the filename, depending on the asset.
 
 **Assets**
 The thumbnail for each scene in a *scenes.json* file can be downloaded with
@@ -165,11 +172,5 @@ The thumbnail for each scene in a *scenes.json* file can be downloaded with
 ```
 The thumbnails will be saved using a directory and filename according to the `datadir` and `filename` options, and will also have a '_thumbnail` suffix. When thumbnails are downloaded an ESRI Worldfile (.wld) file is created, which is a sidecar file that describes the coordinates and resolution of the images. This enables the thumbnails to be viewed in a GIS program like QGIS in their proper geographical location. The world file does not set the spatial reference system used (lat/lon, or WGS-84, or EPSG:4326), so when opened in QGIS it will need to be selected (EPSG:4326).
 
-## Library
-
-The sat-search library is made up of several Python classes. The *Scene* class represents a single set of images for an indentical date (or daterange) and footprint. The *Scenes* class is a collection of *Scene* objects that makes it easier to iterate through them and perform common tasks over all the scenes, such as downloading data.
-
-The *Query* class is a single set of arguments for searching scenes, functions for querying the API with those arguments (and handling of multiple pages if needed) as well storing the results. The higher level *Search* class which is more often used, can deal with multiple *Query* objects, such as individual Scene ids or disparate date ranges that must be issued to the API with different arguments.
-
 ## About
 sat-search was created by [Development Seed](<http://developmentseed.org>) and is part of a collection of tools called [sat-utils](https://github.com/sat-utils).

requirements.txt

@@ -1 +1 @@
-requests~=2.19
+sat-stac~=0.1.0rc5

satsearch/__init__.py

@@ -1,5 +1,4 @@
 from satsearch.search import Search
-from satsearch.scene import Scene, Scenes
 
 import logging
 

satsearch/config.py

@@ -1,7 +1,7 @@
 import os
 
 # API URL
-API_URL = os.getenv('SATUTILS_API_URL', 'https://sat-api.developmentseed.org')
+API_URL = os.getenv('SATUTILS_API_URL', 'https://sat-api-dev.developmentseed.org')
 
 # data directory to store downloaded imagery
 DATADIR = os.getenv('SATUTILS_DATADIR', './${eo:platform}/${date}')

satsearch/main.py

@@ -2,40 +2,41 @@
 import sys
 import json
 from .version import __version__
-from satsearch import Search, Scenes
+from satsearch import Search
+from satstac import Items
 from satsearch.parser import SatUtilsParser
 
 
-def main(scenes=None, print_md=None, print_cal=False,
+def main(items=None, print_md=None, print_cal=False,
          save=None, download=None, **kwargs):
     """ Main function for performing a search """
-    if scenes is None:
-        # get scenes from search
+    if items is None:
+        # get items from search
         search = Search(**kwargs)
-        scenes = Scenes(search.scenes(), properties=kwargs)
+        items = search.items()
     else:
-        scenes = Scenes.load(scenes)
+        items = Items.load(items)
 
     # print metadata
     if print_md is not None:
-        scenes.print_scenes(print_md)
+        items.print_summary(print_md)
 
     # print calendar
     if print_cal:
-        print(scenes.text_calendar())
+        print(items.text_calendar())
 
     # save all metadata in JSON file
     if save is not None:
-        scenes.save(filename=save)
+        items.save(filename=save)
 
-    print('%s scenes found' % len(scenes))
+    print('%s items found' % len(items))
 
     # download files given keys
     if download is not None:
         for key in download:
-            scenes.download(key=key)
+            items.download(key=key)
 
-    return scenes
+    return items
 
 
 def cli():
@@ -50,7 +51,7 @@ def cli():
 
     cmd = args.pop('command', None)
     if cmd is not None:
-        main(**args)
+        return main(**args)
 
 
 if __name__ == "__main__":

satsearch/parser.py

@@ -3,7 +3,10 @@
 import sys
 import logging
 import argparse
+
 import satsearch.config as config
+
+from satstac.utils import dict_merge
 from .version import __version__
 
 
@@ -58,7 +61,24 @@ def parse_args(self, *args, **kwargs):
             config.DATADIR = args.pop('datadir')
         if 'filename' in args:
             config.FILENAME = args.pop('filename')
-
+
+        if 'property' in args:
+            queries = {}
+            for p in args['property']:
+                symbols = {
+                    '=': 'eq',
+                    '>': 'gt',
+                    '<': 'lt',
+                    '>=': 'gte',
+                    '<=': 'lte'
+                }
+                for s in symbols:
+                    parts = p.split(s)
+                    if len(parts) == 2:
+                        queries = dict_merge(queries, {parts[0]: {symbols[s]: parts[1]}})
+                        break
+            args['query'] = queries
+            del args['property']
         return args
 
     @classmethod
@@ -68,26 +88,27 @@ def newbie(cls, *args, **kwargs):
         subparser = parser.add_subparsers(dest='command')
         parents = [parser.pparser, parser.output_parser]
 
-        sparser = subparser.add_parser('search', help='Perform new search of scenes', parents=parents)
+        sparser = subparser.add_parser('search', help='Perform new search of items', parents=parents)
         """ Adds search arguments to a parser """
         parser.search_group = sparser.add_argument_group('search options')
-        parser.search_group.add_argument('-c', '--c:id', help='Name(s) of collection', nargs='*', default=None)
+        parser.search_group.add_argument('-c', '--collection', help='Name of collection', default=None)
+        parser.search_group.add_argument('--bbox', help='Bounding box (min lon, min lat, max lon, max lat)', nargs=4)
         parser.search_group.add_argument('--intersects', help='GeoJSON Feature (file or string)')
+        parser.search_group.add_argument('--datetime', help='Single date/time or begin and end date/time (e.g., 2017-01-01/2017-02-15)')
+        parser.search_group.add_argument('--sort', help='Sort by fields')
         #group.add_argument('--id', help='One or more scene IDs', nargs='*', default=None)
         #group.add_argument('--contains', help='lon,lat points')
-        parser.search_group.add_argument('--datetime', help='Single date/time or begin and end date/time (e.g., 2017-01-01/2017-02-15')
-        parser.search_group.add_argument('--eo:cloud_cover', help='Range of acceptable cloud cover (e.g., 0/20)')
-        parser.search_group.add_argument('-p', '--param', nargs='*', help='Additional parameters of form KEY=VALUE', action=SatUtilsParser.KeyValuePair)
+        parser.search_group.add_argument('-p', '--property', nargs='*', help='Properties of form KEY=VALUE (<, >, <=, >=, = supported)')
         parser.search_group.add_argument('--url', help='URL of the API', default=config.API_URL)
 
         parents.append(parser.download_parser)
-        lparser = subparser.add_parser('load', help='Load scenes from previous search', parents=parents)
-        lparser.add_argument('scenes', help='GeoJSON file of scenes')
+        lparser = subparser.add_parser('load', help='Load items from previous search', parents=parents)
+        lparser.add_argument('items', help='GeoJSON file of Items')
         return parser
 
     class KeyValuePair(argparse.Action):
         """ Custom action for getting arbitrary key values from argparse """
         def __call__(self, parser, namespace, values, option_string=None):
             for val in values:
                 n, v = val.split('=')
-                setattr(namespace, n, v)
+                setattr(namespace, n, {'eq': v})