Add workflow to gen data #329
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,10 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -e | ||
|
|
||
| pip install --upgrade pip | ||
| #pip install 'urllib3[secure]' | ||
| pip install -r requirements.txt | ||
| pip install -r download/requirements.txt | ||
| sudo gem install pg bundler | ||
| sudo bundle install |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,103 @@ | ||
| name: "Generate Website Data" | ||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| jobs: | ||
|
There was a problem hiding this comment. Do we really want this to run on every single push? Do we have unlimited GH Actions hours? There was a problem hiding this comment. I have not heard of limited hours. There is a concept of throttling. So if we are a highly active bunch, that may lead to problems. Otherwise, we're ok. I also have some checks to limit what is run. If we want to, we can limit jobs/steps based on the files that were changed. For now, I'm keeping it mostly simple, except for the piece that builds the docker image (which only occurs when required). There was a problem hiding this comment. Found this, "GitHub Actions usage is free for standard GitHub-hosted runners in public repositories". It's amazing how much compute & storage GH gives away for free 😄 |
||
| build: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| devcontainer: ${{ steps.filter.outputs.devcontainer }} | ||
| noncontainer: ${{ steps.filter.outputs.noncontainer }} | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - name: Get changed files | ||
| id: changed-files | ||
| uses: tj-actions/changed-files@v40 | ||
| - name: List all changed files | ||
| id: filter | ||
| run: | | ||
| devcontainer=false | ||
| noncontainer=true | ||
| for file in ${{ steps.changed-files.outputs.all_changed_files }}; do | ||
| echo "$file was changed" | ||
| if [[ ${{github.event_name}} = push ]]; then | ||
| if [[ $file = .devcontainer* ]]; then | ||
| devcontainer=true | ||
| elif [[ $file = *requirements.txt* ]]; then | ||
| devcontainer=true | ||
| elif [[ $file = Gemfile* ]]; then | ||
| devcontainer=true | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| echo "devcontainer=$devcontainer" >> $GITHUB_OUTPUT | ||
| echo "noncontainer=$noncontainer" >> $GITHUB_OUTPUT | ||
| - name: Login to GitHub Container Registry | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{github.actor}} | ||
| password: ${{secrets.GITHUB_TOKEN}} | ||
| - name: Build dev container | ||
| if: steps.filter.outputs.devcontainer == 'true' | ||
| run: | | ||
| docker build --no-cache --tag ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest -f ./.devcontainer/Dockerfile . | ||
| docker push ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest | ||
| - name: Check code changes | ||
|
There was a problem hiding this comment. Are tags immutable on GHCR? If not, why not use github.ref_name as the tag instead of a separate "sub-repo" for every commit? There was a problem hiding this comment. If I understand your first question, I think you're asking why we even use 'latest' instead of just using the branch name. The reason is mainly convention. Typically, there's a version number or string, but I don't want to generate a new version on every commit. I just want to replace the latest. "GitHub Packages usage is free for public packages." I don't think there's a limit and they actually don't let you delete it if it's popular. Also, we are creating an image per branch and not per commit. It gets overwritten per branch. There was a problem hiding this comment. It's a small thing, but to clarify my first question, in the form |
||
| if: steps.filter.outputs.noncontainer == 'true' | ||
| run: | | ||
| echo "TODO: run test to verify that code changes are good" | ||
| generate: | ||
| needs: build | ||
| if: needs.build.outputs.noncontainer == 'true' | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: ghcr.io/caciviclab/disclosure-backend-static/${{github.ref_name}}:latest | ||
| credentials: | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.github_token }} | ||
| env: | ||
| REPO_OWNER: ${{ github.repository_owner}} | ||
| REPO_BRANCH: ${{ github.ref_name }} | ||
| SERVICE_ACCOUNT_KEY_JSON: ${{ secrets.SERVICE_ACCOUNT_KEY_JSON }} | ||
| GDRIVE_FOLDER: ${{ vars.GDRIVE_FOLDER }} | ||
| PGHOST: postgres | ||
| PGDATABASE: disclosure-backend | ||
| PGUSER: app_user | ||
| PGPASSWORD: app_password | ||
| services: | ||
| postgres: | ||
| #image: postgres:9.6-bullseye | ||
| image: postgres:15.6-bullseye | ||
|
There was a problem hiding this comment. Remove commented-out line There was a problem hiding this comment. I'm trying to remind myself that the postgres versions don't match up with travis-ci and so we have migrate travis-ci. I think I ran into some problem trying to set up 9.6. There was a problem hiding this comment. Gotcha. Maybe a note about it n the comment then? |
||
| env: | ||
| POSTGRES_USER: app_user | ||
| POSTGRES_DB: disclosure-backend | ||
|
There was a problem hiding this comment. Would be nice to have these in top-level env vars in the workflow so they can be shared between the two places they get used. |
||
| POSTGRES_PASSWORD: app_password | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Check setup | ||
| run: | | ||
| git -v | ||
| # This keeps git from thinking that the current dir is not a repo even though a .git dir exists | ||
| git config --global --add safe.directory "$GITHUB_WORKSPACE" | ||
| psql -l | ||
| echo "c1,c2" > test.csv | ||
| echo "a,b" >> test.csv | ||
| cat test.csv | ||
| csvsql -v --db postgresql:///disclosure-backend --insert test.csv | ||
| echo "List tables" | ||
| psql -c "SELECT * FROM pg_catalog.pg_tables WHERE schemaname != 'pg_catalog' AND schemaname != 'information_schema';" | ||
|
|
||
| pip show sqlalchemy | ||
| - name: Create csv files | ||
| run: | | ||
| make clean | ||
| make download | ||
| make import | ||
| make process | ||
| - name: Summarize results | ||
| run: | | ||
| echo "List tables" | ||
| psql -c "SELECT * FROM pg_catalog.pg_tables WHERE schemaname != 'pg_catalog' AND schemaname != 'information_schema';" | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,4 +18,6 @@ cat <<-QUERY | psql ${database_name} | |
| DELETE FROM "$table_name" | ||
| WHERE "Tran_Date" is NULL; | ||
| QUERY | ||
| else | ||
| echo | ||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do this? I can't see why the docker build would need to
pwdThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed