-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add workflow to gen data #329
base: feat/pull-v2-api
Are you sure you want to change the base?
Conversation
This is an automated update by travis-ci at Sat Apr 13 17:13:22 PDT 2024 [skip ci]
11fdf8e
to
5769d8b
Compare
This is an automated update by travis-ci at Tue May 28 21:29:40 PDT 2024 [skip ci]
Build diff from Commit 1e99f65:diff --git a/build/_data/elections/oakland/2018-06-05.json b/build/_data/elections/oakland/2018-06-05.json
index b39b338..140316d 100644
--- a/build/_data/elections/oakland/2018-06-05.json
+++ b/build/_data/elections/oakland/2018-06-05.json
@@ -57,7 +57,7 @@
"total_contributions": 15000.0
},
{
- "name": "Service Employees International Union Local 1021 Issues PAC",
+ "name": "Oakland Athletics Baseball Company",
"type": "Measure",
"election_name": "oakland-june-2018",
"total_contributions": 10000.0
diff --git a/build/_data/elections/oakland/2023-11-07.json b/build/_data/elections/oakland/2023-11-07.json
index 4e77c51..1e5045c 100644
--- a/build/_data/elections/oakland/2023-11-07.json
+++ b/build/_data/elections/oakland/2023-11-07.json
@@ -73,19 +73,19 @@
],
"top_contributors_for_offices": [
{
- "name": "UA Local 342",
+ "name": "Service Employees International Union Local 1021 Candidate PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Service Employees International Union Local 1021 Candidate PAC",
+ "name": "Oakland Education Association PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Peralta Federation of Teachers COPE",
+ "name": "Families in Action for Justice",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
diff --git a/build/_data/elections/oakland/2024-03-05.json b/build/_data/elections/oakland/2024-03-05.json
index db9ca88..a6261f2 100644
--- a/build/_data/elections/oakland/2024-03-05.json
+++ b/build/_data/elections/oakland/2024-03-05.json
@@ -48,13 +48,13 @@
],
"top_contributors_for_offices": [
{
- "name": "Steven Von Stade",
+ "name": "Monique C Houston",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
},
{
- "name": "Monique C Houston",
+ "name": "Steven Von Stade",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
diff --git a/build/_data/elections/oakland/2024-11-05.json b/build/_data/elections/oakland/2024-11-05.json
index 8440b5b..8bd5599 100644
--- a/build/_data/elections/oakland/2024-11-05.json
+++ b/build/_data/elections/oakland/2024-11-05.json
@@ -79,19 +79,19 @@
],
"top_contributors_for_offices": [
{
- "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+ "name": "International Association of Firefighters Local 55",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "International Association of Firefighters Local 55",
+ "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "Weylin White",
+ "name": "Todd Scanlin",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 1800.0
diff --git a/build/_data/stats.json b/build/_data/stats.json
index 74f4147..35aa4a8 100644
--- a/build/_data/stats.json
+++ b/build/_data/stats.json
@@ -1,3 +1,3 @@
{
- "date_processed": "2024-04-13 00:08:40 -0700"
+ "date_processed": "2024-05-29 00:08:29 -0700"
}
diff --git a/build/_data/totals.json b/build/_data/totals.json
index f6a1f4e..f82e1c3 100644
--- a/build/_data/totals.json
+++ b/build/_data/totals.json
@@ -1032,7 +1032,7 @@
"total_contributions": 15000.0
},
{
- "name": "Service Employees International Union Local 1021 Issues PAC",
+ "name": "Oakland Athletics Baseball Company",
"type": "Measure",
"election_name": "oakland-june-2018",
"total_contributions": 10000.0
@@ -1385,19 +1385,19 @@
],
"top_contributors_for_offices": [
{
- "name": "UA Local 342",
+ "name": "Service Employees International Union Local 1021 Candidate PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Service Employees International Union Local 1021 Candidate PAC",
+ "name": "Oakland Education Association PAC",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
},
{
- "name": "Peralta Federation of Teachers COPE",
+ "name": "Families in Action for Justice",
"type": "Office",
"election_name": "oakland-2023",
"total_contributions": 1200.0
@@ -1500,13 +1500,13 @@
],
"top_contributors_for_offices": [
{
- "name": "Steven Von Stade",
+ "name": "Monique C Houston",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
},
{
- "name": "Monique C Houston",
+ "name": "Steven Von Stade",
"type": "Office",
"election_name": "oakland-march-2024",
"total_contributions": 600.0
@@ -1646,19 +1646,19 @@
],
"top_contributors_for_offices": [
{
- "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
+ "name": "International Association of Firefighters Local 55",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "International Association of Firefighters Local 55",
+ "name": "International Federation of Professional and Technical Engineers-Local 21 TJ Anthony PAC Fund",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 2400.0
},
{
- "name": "Weylin White",
+ "name": "Todd Scanlin",
"type": "Office",
"election_name": "oakland-2024",
"total_contributions": 1800.0 |
This is an automated update by travis-ci at Wed May 29 19:54:16 PDT 2024 [skip ci]
This is an automated update by travis-ci at Wed May 29 21:15:13 PDT 2024 [skip ci]
…losure-backend-static into add-workflow-to-gen-data
This is an automated update by travis-ci at Wed May 29 21:25:38 PDT 2024 [skip ci]
…losure-backend-static into add-workflow-to-gen-data
@@ -8,3 +8,11 @@ ENV PYTHONUNBUFFERED 1 | |||
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \ | |||
&& apt-get -y install --no-install-recommends postgresql-client ruby-full | |||
|
|||
RUN pwd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do this? I can't see why the docker build would need to pwd
@@ -26,8 +26,10 @@ services: | |||
network_mode: service:db | |||
|
|||
db: | |||
#image: postgres:16.0-bullseye |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove commented-out lines
@@ -4,7 +4,7 @@ | |||
"service": "app", | |||
"workspaceFolder": "/workspace", | |||
"remoteUser": "vscode", | |||
"postCreateCommand": "bash ./.devcontainer/post-create-command.sh", | |||
//"postCreateCommand": "bash ./.devcontainer/post-create-command.sh", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove commented-out line
PGPASSWORD: app_password | ||
services: | ||
postgres: | ||
#image: postgres:9.6-bullseye |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove commented-out line
#image: postgres:9.6-bullseye | ||
image: postgres:15.6-bullseye | ||
env: | ||
POSTGRES_USER: app_user |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to have these in top-level env vars in the workflow so they can be shared between the two places they get used.
name: "Generate Website Data" | ||
on: | ||
workflow_dispatch: | ||
push: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really want this to run on every single push? Do we have unlimited GH Actions hours?
if: steps.filter.outputs.devcontainer == 'true' | ||
run: | | ||
docker build --no-cache --tag ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest -f ./.devcontainer/Dockerfile . | ||
docker push ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are tags immutable on GHCR? If not, why not use github.ref_name as the tag instead of a separate "sub-repo" for every commit?
Does GitHub let us fill their Docker registry with as many repos and tags as we care to? There's no limit? I'm surprised they'd be willing to host an image for every single commit, free of charge. Do the Docker images get pruned after some time or something?
@@ -0,0 +1,15 @@ | |||
CREATE TABLE committees_v2 ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I appreciate that we're proactively defining these tables now instead of relying on csvsql to automagically do it for us.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more comment
@@ -4,15 +4,21 @@ on: | |||
jobs: | |||
check: | |||
runs-on: ubuntu-latest | |||
container: | |||
image: ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens here if no image has been pushed for github.ref_name
?
The purpose of this pull request is to add a GitHub workflow for the ETL that runs in Travis CI. This workflow will be used to help integrate the download logic for netfile v2 into
make download
.The csvkit that was being used was a beta version (0.9.0) and it was making it difficult to upgrade the ETL logic. I updated csvkit to the latest 1.3.0 version in another branch (PR) to address upgrade issues and merged it to this branch. The change includes making use of SQL files for creating the tables. These SQL files originated from csvsql 0.9.0. These files make the schema independent of the version of csvsql used and can be modified going forward as needed to change the schema of the tables.