removing need for Makefile to install elasticsearch dependencies

- installing via setuptools can now handle installing elasticsearch 
dependencies (v1 or v6)
- updating docs to reflect how to install via this method

Author: ryandeivert

.gitignore

@@ -8,5 +8,3 @@ htmlcov/
 config.yaml
 account-data/
 my_account_iam.json
-requirements-dev.es[0-9].txt
-requirements.es[0-9].txt

Makefile

@@ -15,6 +15,8 @@ python3 -m venv ./venv && source venv/bin/activate
 pip install git+https://github.com/duo-labs/cloudtracker.git#egg=cloudtracker
 ```
 
+Note: To install with ElasticSearch support, see the [ElasticSearch docs](docs/elasticsearch.md).
+
 ### Step 2: Download your IAM data
 Download a copy of the IAM data of an account using the AWS CLI:
 

README.md

@@ -402,7 +402,16 @@ def run(args, config, start, end):
     account = get_account(config['accounts'], args.account)
 
     if 'elasticsearch' in config:
-        from cloudtracker.datasources.es import ElasticSearch
+        try:
+            from cloudtracker.datasources.es import ElasticSearch
+        except ImportError:
+            exit(
+                "Elasticsearch support not installed. Install with support via "
+                "'pip install git+https://github.com/duo-labs/cloudtracker.git#egg=cloudtracker[es1]' for "
+                "elasticsearch 1 support, or "
+                "'pip install git+https://github.com/duo-labs/cloudtracker.git#egg=cloudtracker[es6]' for "
+                "elasticsearch 6 support"
+            )
         datasource = ElasticSearch(config['elasticsearch'], start, end)
     else:
         logging.debug("Using Athena")

cloudtracker/__init__.py

@@ -5,8 +5,6 @@ Requirements
 * CloudTrail logs must be loaded into ElasticSearch.  For instructions on setting up ElasticSearch and ingesting an archive of CloudTrail logs into it see below.
   * ElasticSearch 6.x is supported, but there are reports of ElasticSearch 1.x being used successfully.
 
-
-
 Installation
 ============
 
@@ -15,16 +13,15 @@ Install the Python libraries using one of the provided Makefile targets:
 
 For elasticsearch v6.x:
 ```
-make dev_elasticsearchv6
-source venv/bin/activate
+python3 -m venv ./venv && source venv/bin/activate
+pip install git+https://github.com/duo-labs/cloudtracker.git#egg=cloudtracker[es6]
 ```
 
 For older versions, such as elasticsearch v1.x:
 ```
-make dev_elasticsearchv1
-source venv/bin/activate
+python3 -m venv ./venv && source venv/bin/activate
+pip install git+https://github.com/duo-labs/cloudtracker.git#egg=cloudtracker[es1]
 ```
-The target will create a virtualenv in `./venv` and pip install the relevant requirements.
 
 ### Step 2
 Get the IAM data of the account
@@ -188,7 +185,7 @@ cmake -DCMAKE_BUILD_TYPE=release -DEXT_aws=off -DEXT_kafka=off -DEXT_parquet=off
 make
 make packages
 sudo make install
-# In my experience I needed to manually install files, or copy or link them, as you should have files named 
+# In my experience I needed to manually install files, or copy or link them, as you should have files named
 # `rjson.so` and `ltn12.lua` at `/usr/local/lib/luasandbox/io_modules/`.
 ```
 

docs/elasticsearch.md

@@ -34,13 +34,13 @@ def get_version():
     test_suite='tests/unit',
     tests_require=TESTS_REQUIRE,
     extras_require={
-        'dev': TESTS_REQUIRE + ['autoflake', 'autopep8', 'pylint']
+        'dev': TESTS_REQUIRE + ['autoflake', 'autopep8', 'pylint'],
+        'es1': ['elasticsearch==1.9.0', 'elasticsearch_dsl==0.0.11'],
+        'es6': ['elasticsearch==6.1.1', 'elasticsearch_dsl==6.1.0']
     },
     install_requires=[
         'ansicolors==1.1.8',
         'boto3==1.5.32',
-        'elasticsearch==6.1.1',
-        'elasticsearch_dsl==6.1.0',
         'jmespath==0.9.3',
         'pyyaml==3.12'
     ],