Skip to content

Latest commit

 

History

History
166 lines (126 loc) · 4.9 KB

OVERVIEW.md

File metadata and controls

166 lines (126 loc) · 4.9 KB
Raw

Overview

For each each plugin, as identified by a search for pytest-* on PyPI, we download, install and execute its tests using tox. If a plugin doesn't have a tox.ini file, we generate a simple tox.ini which just ensures that the plugin was installed successfully by running pytest --help.

Once we have tested all plugins, the results are posted to a web application that can be used to visualize them.

The steps above are executed for some Python and pytest versions, resulting in a matrix of plugin x Python x pytest compatibility.

Travis is used to execute the tests and post the results. The web page is hosted by heroku at http://plugincompat.herokuapp.com.

Updating

A heroku scheduler job runs once a day that executes the heroku_update.bash script.

The main job of this script is to run update_index.py, which is responsible for querying PyPI and updating index.json with latest pytest plugin information. This file is then pushed back to GitHub which will trigger a new travis build.

Details

Below there's a more detailed description of the system for those interested.

update_index.py

This script creates/updates the file index.json file using new information from PyPI and contains the list of plugins to test. It is a JSON formatted file containing a list of (plugin name, version, description), like this:

[
  {
    "version": "1.3.7",
    "name": "pytest-allure-adaptor",
    "description": "Plugin for pytest to generate allure xml reports"
  },
  {
    "version": "2.1.0",
    "name": "pytest-bdd",
    "description": "BDD for pytest"
  },
  {
    "version": "0.0.1",
    "name": "pytest-beds",
    "description": "Fixtures for testing Google Appengine (GAE) apps"
  }
]

To run the script, just execute it without parameters:

python update_index.py
index.json updated, push to GitHub.

If index.json was updated it means either new plugins were posted or some of the existing plugins were updated, so it should be committed and pushed to GitHub as the message says.

If nothing has changed, no further action is needed:

python update_index.py
index.json skipped, no changes.

run.py

This script should be executed by travis: it reads index.json file, executes tests for each package in the current Python interpreter and posts results back to heroku.

It does so by downloading the source package for each plugin and extracts it into the current directory. It is assumed that plugins use tox for testing; if a plugin doesn't have a tox.ini file, the script will generate a simple tox.ini that just tries to ensure the plugins installs cleanly.

After all plugins are tested, results are posted to the web page.

The script is configured by two environment variables:

PYTEST_VERSION: pytest version that will be passed to tox as --force-dep parameter, ensuring that it is tested against that pytest version and not what is installed in the system.

PLUGINCOMPAT_SITE: URL to post the test result data to. See below for an example of a payload.

The above environment variables are configured in the .travis.yml file and are part of the travis build matrix.

web.py

This is the webserver that is hosted at heroku.

It serves an index page containing a table displaying test results for pytest plugins against different Python and pytest versions.

It supports the following URLs:

Main page

GET /

Returns main page, showing the test results table.

Posting test results

POST /

Posts a new payload that update test results. This payload is generated by our travis build matrix. It expects a secret which is enabled as a Travis environment variable and must be part of the post.

Example of an expected payload:

{
  "secret": "SECRET",
  "results": [
    {
      "name": "pytest-blockage",
      "version": "0.1",
      "env": "py33",
      "pytest": "2.5.2",
      "status": "ok",
      "output": "GLOB sdist-make: /home/travis/...",
      "description": "Disable network requests during a test run."
    }
  ]
}

Status images help

GET /status

Returns a page explaining on how to obtain status images (badges) for each plugin.

Status images for plugins

GET /status/:name/`

Returns an image for a specific plugin indicating its status when tested against a Python and pytest versions. For example: /status/pytest-pep8-1.0.5?py=py33&pytest=2.4.2

web.py has test cases to ensure pages are behaving as expected, see test_web.py.

Get test output

GET /output/:name/

Receives the same parameter as /status/:name/, but returns the output of the tox run as plain text.