Skip to content

Latest commit

 

History

History
69 lines (50 loc) · 2.11 KB

README.md

File metadata and controls

69 lines (50 loc) · 2.11 KB
Raw

Openstates Realtime Lambda

Overview

This is a lambda function that is triggered by an upload of outputs from the openstates-scrapers to an S3 bucket. The lambda function then parses the file and inserts the data into a postgres database. The lambda function is written in python and uses the django library to connect to the database. The lambda function is deployed using Zappa.

Setup

Run the following commands to setup the project

poetry install

ZAPPA BUG NOTE

Please note that right now setuptools is pinned to an old version to mitigate a Zappa bug that was very recently fixed, but seems to not be released in any easy-to-poetry-install way. Bug details here.

Deployment

Deployment depends on setup above, as well as having the AWS CLI installed and a login profile for the CLI that has access to the Open States AWS project. You will use AWS_DEFAULT_PROFILE in commands below to indicate the name of that login profile. In these examples it will be openstates.

For the first deployment, run the following command

AWS_DEFAULT_PROFILE=openstates poetry run zappa deploy [stage]

For subsequent deployments, run the following command

AWS_DEFAULT_PROFILE=openstates poetry run zappa update [stage]

Where [stage] is the stage to deploy to. This can be either dev or prod.

Deploy via Docker

  • Make a copy of Dockerfile-example to create Dockerfile
  • Update AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your creds
  • Run docker build --progress=plain .

S3 Bucket Lifecyle

The S3 bucket that the lambda function is uses should have a lifecycle policy that deletes the files after 2 days. The CLI command for this is below:

aws s3api put-bucket-lifecycle-configuration --bucket openstates-realtime-bills --lifecycle-configuration '{
  "Rules": [
    {
      "Status": "Enabled",
      "Filter": {
        "Prefix": "archive/"
      },
      "Expiration": {
        "Days": 2
      },
      "ID": "Delete after 2 days"
    }
  ]
}'