Skip to content

Latest commit

 

History

History
42 lines (28 loc) · 1.75 KB

README.md

File metadata and controls

42 lines (28 loc) · 1.75 KB
Raw

Public Ethics Data Downloader CircleCI codecov

This repository contains scripts used to download ethics data for the City of Oakland.

We currently support retrieving the 2018-2019 FPPC Form 700 Statement of Economic Interests. The data is downloaded from Netfile and uploaded to a data warehouse.

Usage

The logic to download and parse filings is normally deployed to Google Cloud as cloud functions.

The logic can also be run locally as scripts in the scripts directory. download_form_700_data.py will download all filings to scripts/filings. parse_local_data.py will extract data from the downloaded files to a SQLite database.

These scripts can be run with a command like the one below:

python -m scripts.download_form_700_data

Development

We use pipenv to manage environments and requirements, so install that first.

  1. Install the requirements for this project:

    make requirements

  2. Run code quality inspection:

    make quality

  3. Run tests:

    make test

Deployment

This code is deployed to Google Cloud as functions responsible for (a) downloading filings and (b) performing ETL. The functions can all be deployed by running ./deploy.sh.