Linter and requirements updates (#575)

* Use black, flake8, and isort instead of just flake8
* Reorganize requirements for simplicity
* Update some rules
* Add a script for automatically fixing linter errors
* Add some docs on linting
* Run travis linter on python 3.6

.linter-files

@@ -0,0 +1 @@
+mozregression tests setup.py gui/mozregui gui/build.py gui/tests

.travis.yml

@@ -7,15 +7,24 @@ matrix:
       dist: trusty # newer dists seem to have some weird ssl error with python2
       language: python
       python: '2.7'
-      virtualenv:
-        system_site_packages: true
       script:
+        - pip install -e .
         - coverage run -m pytest tests && mv .coverage .coverage.core
         - coverage combine
         - pip install coveralls; coveralls
-    - name: python3-linux
+    - name: python35-linux
       env: PYTHON=python3.5
       os: linux
+      language: python
+      python: '3.5'
+      script:
+        - pip install -e .
+        - coverage run -m pytest tests && mv .coverage .coverage.core
+        - coverage combine
+        - pip install coveralls; coveralls
+    - name: python3-linux
+      env: PYTHON=python3.6
+      os: linux
       addons:
         apt:
           packages:
@@ -24,22 +33,22 @@ matrix:
       services:
         - xvfb
       language: python
-      python: '3.5'
+      python: '3.6'
       script:
-        - pip install -r requirements-gui-dev.txt
+        - pip install -r requirements/all.txt
         - coverage run -m pytest tests && mv .coverage .coverage.core
         - coverage run gui/build.py test && mv .coverage .coverage.gui
         - coverage combine
         - pip install coveralls; coveralls
         - python gui/build.py bundle
     - name: linters
-      env: PYTHON=python3.5
+      env: PYTHON=python3.7
       os: linux
       language: python
-      python: '3.5'
+      python: '3.7'
       script:
-        - flake8 mozregression tests setup.py
-        - flake8 gui/mozregui gui/build.py gui/tests
+        - pip install -r requirements/linters.txt
+        - ./bin/lint-check.sh || (echo "Lint fix results:" && ./bin/lint-fix.sh && git diff && false)
 
 install:
     - if [ "$TRAVIS_OS_NAME" == "osx" ]; then
@@ -56,7 +65,7 @@ install:
           sudo install_name_tool -id $PWD/QtGui.so QtGui.so;
           cd $MOZPATH;
       fi
-    - pip install -r requirements-dev.txt
+    - pip install -r requirements/console.txt
 
 deploy:
   - provider: releases

README.md

@@ -3,20 +3,21 @@
 mozregression is an interactive regression rangefinder for quickly tracking down the source of bugs in Mozilla nightly and inbound builds.
 
 You can start using mozregression today:
-- [start with our installation guide](https://mozilla.github.io/mozregression/install.html), then 
-- take a look at [our Quick Start document](https://mozilla.github.io/mozregression/quickstart.html). 
+
+- [start with our installation guide](https://mozilla.github.io/mozregression/install.html), then
+- take a look at [our Quick Start document](https://mozilla.github.io/mozregression/quickstart.html).
 
 ## Status
 
 [![Latest Version](https://img.shields.io/pypi/v/mozregression.svg)](https://pypi.python.org/pypi/mozregression/)
 [![License](https://img.shields.io/pypi/l/mozregression.svg)](https://pypi.python.org/pypi/mozregression/)
 
-
 Build status:
- - Linux:
-   [![Linux Build Status](https://travis-ci.org/mozilla/mozregression.svg?branch=master)](https://travis-ci.org/mozilla/mozregression)
-   [![Coverage Status](https://img.shields.io/coveralls/mozilla/mozregression.svg)](https://coveralls.io/r/mozilla/mozregression)
- - Windows: [![Windows Build status](https://ci.appveyor.com/api/projects/status/bcg7t1pt2bahggdr?svg=true)](https://ci.appveyor.com/project/wlach/mozregression/branch/master)
+
+- Linux:
+  [![Linux Build Status](https://travis-ci.org/mozilla/mozregression.svg?branch=master)](https://travis-ci.org/mozilla/mozregression)
+  [![Coverage Status](https://img.shields.io/coveralls/mozilla/mozregression.svg)](https://coveralls.io/r/mozilla/mozregression)
+- Windows: [![Windows Build status](https://ci.appveyor.com/api/projects/status/bcg7t1pt2bahggdr?svg=true)](https://ci.appveyor.com/project/wlach/mozregression/branch/master)
 
 For more information see:
 
@@ -41,27 +42,39 @@ This is recommended.
 If you are **really sure** that you only want to hack on the mozregression command line:
 
 1. Install [virtualenv](https://virtualenv.pypa.io/en/stable/)
-  or [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org/en/latest/).
+   or [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org/en/latest/).
 
 2. install dependencies:
 
-  With virtualenvwrapper:
+With virtualenvwrapper:
+
+```bash
+mkvirtualenv -p /usr/bin/python3 mozregression
+pip install -r requirements/all-console.txt
+```
+
+Or with virtualenv: ::
+
+```bash
+virtualenv -p /usr/bin/python3 venv
+source venv/bin/activate
+pip install -r requirements/all-console.txt
+```
+
+2. lint your code for errors and formatting (we use [black](https://black.readthedocs.io/en/stable/), [flake8](https://flake8.pycqa.org/en/latest/) and [isort](https://isort.readthedocs.io/en/latest/))
 
-  ```bash
-  mkvirtualenv -p /usr/bin/python3 mozregression
-  pip install -r requirements-dev.txt
-  ```
+```bash
+./bin/lint-check.sh
+```
 
-  Or with virtualenv: ::
+If it turns up errors, try using the `lint-fix.sh` script to fix any errors which can be addressed automatically:
 
-  ```bash
-  virtualenv -p /usr/bin/python3 venv
-  source venv/bin/activate
-  pip install -r requirements-dev.txt
-  ```
+```bash
+./bin/lint-fix.sh
+```
 
 3. run tests (be sure that your virtualenv is activated):
 
-  ```bash
-  pytest tests
-  ```
+```bash
+pytest tests
+```

appveyor.yml

@@ -8,7 +8,7 @@ init:
   - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
 install:
   # install mozregression code and test dependencies
-  - "pip install -r requirements-gui-dev.txt"
+  - "pip install -r requirements/all.txt"
 test_script:
   - "python setup.py test"
   - "python gui\\build.py test"

bin/lint-check.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+set -e
+
+LINTER_FILES="$(dirname "$0")/../.linter-files"
+
+cat $LINTER_FILES | xargs isort --check-only --recursive
+cat $LINTER_FILES | xargs flake8

bin/lint-fix.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+set -e
+
+LINTER_FILES="$(dirname "$0")/../.linter-files"
+
+cat $LINTER_FILES | xargs isort --recursive -y
+cat $LINTER_FILES | xargs black

gui/README.rst

@@ -23,18 +23,18 @@ a virtualenv here. See this link
 about python virtualenvs. You may also consider using virtualenvwrapper
 (https://virtualenvwrapper.readthedocs.org/en/latest/).
 
-Python 3.5+ is *required* to develop or use mozregression-gui.
+Python 3.6+ is *required* to develop or use mozregression-gui.
 
 Install with virtualenvwrapper: ::
 
    mkvirtualenv -p /usr/bin/python3 mozregression
-   pip install -r requirements-gui-dev.txt
+   pip install -r requirements/all.txt
 
 Or with virtualenv: ::
 
    virtualenv --system-site-packages -p /usr/bin/python3 venv
    source venv/bin/activate
-   pip install -r requirements-gui-dev.txt
+   pip install -r requirements/all.txt
 
 Launching the application
 -------------------------

gui/build.py

@@ -5,109 +5,116 @@
 """
 
 import argparse
-import sys
-import subprocess
-import os
-import shutil
 import glob
+import os
 import pipes
+import shutil
+import subprocess
+import sys
 import tarfile
 
-
-IS_WIN = os.name == 'nt'
-IS_MAC = sys.platform == 'darwin'
+IS_WIN = os.name == "nt"
+IS_MAC = sys.platform == "darwin"
 
 
 def call(*args, **kwargs):
-    print('Executing `%s`' % ' '.join(pipes.quote(a) for a in args))
+    print("Executing `%s`" % " ".join(pipes.quote(a) for a in args))
     subprocess.check_call(args, **kwargs)
 
 
 def py_script(script_name):
     python_dir = os.path.dirname(sys.executable)
     if IS_WIN:
-        return os.path.join(python_dir, 'Scripts',
-                            script_name + '.exe')
+        return os.path.join(python_dir, "Scripts", script_name + ".exe")
     else:
         return os.path.join(python_dir, script_name)
 
 
 def do_uic(options, force=False):
-    for uifile in glob.glob('mozregui/ui/*.ui'):
-        pyfile = os.path.splitext(uifile)[0] + '.py'
-        if force or not os.path.isfile(pyfile) or \
-                (os.path.getmtime(uifile) > os.path.getmtime(pyfile)):
+    for uifile in glob.glob("mozregui/ui/*.ui"):
+        pyfile = os.path.splitext(uifile)[0] + ".py"
+        if (
+            force
+            or not os.path.isfile(pyfile)
+            or (os.path.getmtime(uifile) > os.path.getmtime(pyfile))
+        ):
             print("uic'ing %s -> %s" % (uifile, pyfile))
-            os.system('pyside2-uic {} > {}'.format(uifile, pyfile))
+            os.system("pyside2-uic {} > {}".format(uifile, pyfile))
 
 
 def do_rcc(options, force=False):
-    rccfile = 'resources.qrc'
-    pyfile = 'resources_rc.py'
-    if force or not os.path.isfile(pyfile) or \
-            (os.path.getmtime(rccfile) > os.path.getmtime(pyfile)):
+    rccfile = "resources.qrc"
+    pyfile = "resources_rc.py"
+    if (
+        force
+        or not os.path.isfile(pyfile)
+        or (os.path.getmtime(rccfile) > os.path.getmtime(pyfile))
+    ):
         print("rcc'ing %s -> %s" % (rccfile, pyfile))
-        call('pyside2-rcc', '-o', pyfile, rccfile)
+        call("pyside2-rcc", "-o", pyfile, rccfile)
 
 
 def do_run(options):
     do_uic(options)
     do_rcc(options)
-    call(sys.executable, 'mozregression-gui.py')
+    call(sys.executable, "mozregression-gui.py")
 
 
 def do_test(options):
     do_uic(options)
     do_rcc(options)
-    print('Running tests...')
+    print("Running tests...")
     import pytest
-    sys.exit(pytest.main(['tests', '-v']))
+
+    sys.exit(pytest.main(["tests", "-v"]))
 
 
 def do_bundle(options):
     do_uic(options, True)
     do_rcc(options, True)
 
     # clean previous runs
-    for dirname in ('build', 'dist'):
+    for dirname in ("build", "dist"):
         if os.path.isdir(dirname):
             shutil.rmtree(dirname)
     # create a bundle for the application
-    call('pyinstaller', 'gui.spec')
+    call("pyinstaller", "gui.spec")
     # create an installer
     if IS_WIN:
         makensis_path = os.path.join(options.nsis_path, "makensis.exe")
-        call(makensis_path, 'wininst.nsi', cwd='wininst')
+        call(makensis_path, "wininst.nsi", cwd="wininst")
     elif IS_MAC:
-        call('hdiutil', 'create', 'dist/mozregression-gui.dmg',
-             '-srcfolder', 'dist/', '-ov')
+        call(
+            "hdiutil", "create", "dist/mozregression-gui.dmg", "-srcfolder", "dist/", "-ov",
+        )
     else:
-        with tarfile.open('mozregression-gui.tar.gz', 'w:gz') as tar:
-            tar.add(r'dist/')
+        with tarfile.open("mozregression-gui.tar.gz", "w:gz") as tar:
+            tar.add(r"dist/")
 
 
 def parse_args():
     parser = argparse.ArgumentParser()
     subparsers = parser.add_subparsers()
 
-    uic = subparsers.add_parser('uic', help='build uic files')
+    uic = subparsers.add_parser("uic", help="build uic files")
     uic.set_defaults(func=do_uic)
 
-    rcc = subparsers.add_parser('rcc', help='build rcc files')
+    rcc = subparsers.add_parser("rcc", help="build rcc files")
     rcc.set_defaults(func=do_rcc)
 
-    run = subparsers.add_parser('run', help='run the application')
+    run = subparsers.add_parser("run", help="run the application")
     run.set_defaults(func=do_run)
 
-    test = subparsers.add_parser('test', help='run the unit tests')
+    test = subparsers.add_parser("test", help="run the unit tests")
     test.set_defaults(func=do_test)
 
-    bundle = subparsers.add_parser('bundle',
-                                   help='bundle the application (freeze)')
+    bundle = subparsers.add_parser("bundle", help="bundle the application (freeze)")
     if IS_WIN:
-        bundle.add_argument('--nsis-path', default='C:\\NSIS',
-                            help='your NSIS path on the'
-                            ' system(default: %(default)r)')
+        bundle.add_argument(
+            "--nsis-path",
+            default="C:\\NSIS",
+            help="your NSIS path on the" " system(default: %(default)r)",
+        )
 
     bundle.set_defaults(func=do_bundle)
 
@@ -122,8 +129,8 @@ def main():
     try:
         options.func(options)
     except Exception as e:
-        sys.exit('ERROR: %s' % e)
+        sys.exit("ERROR: %s" % e)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()