Skip to content

Commit 9a481bd

Browse files
wslulciucwslulciuc
authored
Add CI workflow for testing db migrations (#2326)
* Add workflows/db.yml and wip script for testing db migrations Signed-off-by: wslulciuc <[email protected]> * Add --args to ./docker/up.sh Signed-off-by: wslulciuc <[email protected]> * Add db-data volume Signed-off-by: wslulciuc <[email protected]> * Add .github/workflows/db.sh Signed-off-by: wslulciuc <[email protected]> * continued: Add .github/workflows/db.sh Signed-off-by: wslulciuc <[email protected]> * db.sh -> db-migration.sh Signed-off-by: wslulciuc <[email protected]> * Use git sha1 for tag on `--build` Signed-off-by: wslulciuc <[email protected]> * Use `seed_api` for docker service name and build on `--build` Signed-off-by: wslulciuc <[email protected]> * Remove cd into root dir in docker/prune.sh Signed-off-by: wslulciuc <[email protected]> * Cleanup docker/volumes.sh Signed-off-by: wslulciuc <[email protected]> * Fix log error formatting Signed-off-by: wslulciuc <[email protected]> * Add db-backup volume Signed-off-by: wslulciuc <[email protected]> * Add `--no-web` and `--no-volumes` flags Signed-off-by: wslulciuc <[email protected]> * Add `docker-compose.web-dev.yml` to support `--no-web` flag Signed-off-by: wslulciuc <[email protected]> * Add `log_db_migration()` to `db-migration.sh` Signed-off-by: wslulciuc <[email protected]> * Add comments and cleanup logic in `docker/up.sh` Signed-off-by: wslulciuc <[email protected]> * Use `sudo` to run `db-migration.sh` Signed-off-by: wslulciuc <[email protected]> * Fix logic for `--no-web` and `--no-volumes` Signed-off-by: wslulciuc <[email protected]> * continued: Fix logic for `--no-web` and `--no-volumes` Signed-off-by: wslulciuc <[email protected]> * Remove `-it` flags from log_db_migration() Signed-off-by: wslulciuc <[email protected]> * Set image in docker-compose.web-dev.yml Signed-off-by: wslulciuc <[email protected]> * Use dev compose file in build mode when seeding http api server Signed-off-by: wslulciuc <[email protected]> * Set default value for `--build` Signed-off-by: wslulciuc <[email protected]> * Append args Signed-off-by: wslulciuc <[email protected]> * Cleanup image tags in dev mode Signed-off-by: wslulciuc <[email protected]> * More debugging... Signed-off-by: wslulciuc <[email protected]> * Set `DOCKER_BUILDKIT` and reorder appending compose files Signed-off-by: wslulciuc <[email protected]> * Add migrate-db CI job Signed-off-by: wslulciuc <[email protected]> * Cleanup debugging Signed-off-by: wslulciuc <[email protected]> * Fix path to db-migration.sh Signed-off-by: wslulciuc <[email protected]> * Remove deps for build-api CI job Signed-off-by: wslulciuc <[email protected]> * continued: Remove deps for build-api CI job Signed-off-by: wslulciuc <[email protected]> * Limit output from db-migration.sh Signed-off-by: wslulciuc <[email protected]> * Print entire flyway_schema_history row Signed-off-by: wslulciuc <[email protected]> * Install docker compose in migrate-db CI job Signed-off-by: wslulciuc <[email protected]> * Ensure migrate-db CI jobs fails on migration issues Signed-off-by: wslulciuc <[email protected]> * Order db migrations by installed_on Signed-off-by: wslulciuc <[email protected]> * Update comments Signed-off-by: wslulciuc <[email protected]> * Update chart docs to use docker-compose.db.yml Signed-off-by: wslulciuc <[email protected]> * Rename db migration backup container Signed-off-by: wslulciuc <[email protected]> * Add additional constants and comments to `db-migration.sh` Signed-off-by: wslulciuc <[email protected]> * Apply minor formatting Signed-off-by: wslulciuc <[email protected]> * continued: Apply minor formatting Signed-off-by: wslulciuc <[email protected]> * continued: Apply minor formatting Signed-off-by: wslulciuc <[email protected]> * continued: Apply minor formatting Signed-off-by: wslulciuc <[email protected]> * continued: Apply minor formatting Signed-off-by: wslulciuc <[email protected]> * Always printout docker container logs for debugging Signed-off-by: wslulciuc <[email protected]> * Replace `latest` with `0.29.0` in `db-migration.sh` Signed-off-by: wslulciuc <[email protected]> * Log simple log on migration error Signed-off-by: wslulciuc <[email protected]> * Simplify script output for readability Signed-off-by: wslulciuc <[email protected]> * continued: Simplify script output for readability Signed-off-by: wslulciuc <[email protected]> * Add docs to `db-migration.sh` Signed-off-by: wslulciuc <[email protected]> Signed-off-by: wslulciuc <[email protected]>
1 parent 71ceaab commit 9a481bd

13 files changed

+256
-67
lines changed

‎.circleci/config.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,15 @@ jobs:
148148
- run: npm install --prefix=${HOME}/.local --global redoc-cli
149149
- run: redoc-cli bundle spec/openapi.yml
150150

151+
migrate-db:
152+
working_directory: ~/marquez
153+
machine:
154+
image: ubuntu-2004:current
155+
steps:
156+
- checkout
157+
- run: ./.circleci/get-docker-compose.sh
158+
- run: ./.circleci/db-migration.sh
159+
151160
release-java:
152161
working_directory: ~/marquez
153162
machine:
@@ -195,6 +204,9 @@ workflows:
195204
- unit-test-web
196205
- unit-test-client-python
197206
- lint-spec-api
207+
- migrate-db:
208+
requires:
209+
- build-api
198210
release:
199211
jobs:
200212
- build-client-python:

‎.circleci/db-migration.sh

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/bin/bash
2+
#
3+
# Copyright 2018-2022 contributors to the Marquez project
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# A script used in CI to test database migrations by:
7+
# (1) Applying db migrations on latest Marquez release
8+
# (2) Take a backup of db from Step 1
9+
# (3) Applying db migrations on latest Marquez build using backup
10+
#
11+
# Usage: $ ./db-migration.sh
12+
13+
# Version of PostgreSQL
14+
readonly POSTGRES_VERSION="12.1"
15+
# Version of Marquez
16+
readonly MARQUEZ_VERSION="0.29.0"
17+
# Build version of Marquez
18+
readonly MARQUEZ_BUILD_VERSION="$(git log --pretty=format:'%h' -n 1)" # SHA1
19+
20+
readonly DB_MIGRATION_VOLUME="marquez_db-backup"
21+
readonly DB_MIGRATION_BACKUP="db-migration-backup"
22+
readonly DB_MIGRATION_QUERY=$(cat <<-END
23+
SELECT version,installed_on,checksum
24+
FROM flyway_schema_history
25+
WHERE version IS NOT NULL
26+
ORDER BY installed_on DESC LIMIT 1;
27+
END
28+
)
29+
30+
log() {
31+
echo -e "\033[1m>>\033[0m ${1}"
32+
}
33+
34+
error() {
35+
echo -e "\033[0;31merror: ${1}\033[0m"
36+
}
37+
38+
exit_with_cause() {
39+
log "please view container logs for more details on cause:"
40+
docker-compose logs
41+
exit 1
42+
}
43+
44+
query_db_migration() {
45+
# Start db using backup
46+
[[ $(docker ps -f "name=${DB_MIGRATION_BACKUP}" --format '{{.Names}}') == "${DB_MIGRATION_BACKUP}" ]] || \
47+
docker run -d --name "${DB_MIGRATION_BACKUP}" \
48+
-v "${DB_MIGRATION_VOLUME}:/var/lib/postgresql/data" \
49+
"postgres:${POSTGRES_VERSION}"
50+
# Query applied db migrations
51+
log "latest migration applied to db:"
52+
docker exec "${DB_MIGRATION_BACKUP}" \
53+
psql -U marquez -c "${DB_MIGRATION_QUERY}"
54+
}
55+
56+
# Change working directory to project root
57+
project_root=$(git rev-parse --show-toplevel)
58+
cd "${project_root}/"
59+
60+
# (1) Apply db migrations on latest Marquez release
61+
log "start db with latest migrations (marquez=${MARQUEZ_VERSION}):"
62+
if ! ./docker/up.sh \
63+
--args "--exit-code-from seed_marquez" \
64+
--tag "${MARQUEZ_VERSION}" \
65+
--no-web \
66+
--seed > /dev/null; then
67+
error "failed to start db using backup!"
68+
exit_with_cause
69+
fi
70+
71+
# Query, then display schema migration applied
72+
query_db_migration
73+
74+
# (2) Apply db migrations on latest Marquez build using backup
75+
log "start db using backup (marquez=${MARQUEZ_BUILD_VERSION}):"
76+
if ! ./docker/up.sh \
77+
--args "--exit-code-from seed_marquez" \
78+
--no-web \
79+
--no-volumes \
80+
--build \
81+
--seed > /dev/null; then
82+
error "failed to start db using backup!"
83+
exit_with_cause
84+
fi
85+
86+
# Query, then display additional schema migration applied on backup (if any)
87+
query_db_migration
88+
89+
log "DONE!"

‎.circleci/get-docker-compose.sh

Lines changed: 10 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,18 @@
11
#!/bin/bash
22
#
3-
# Licensed under the Apache License, Version 2.0 (the "License");
4-
# you may not use this file except in compliance with the License.
5-
# You may obtain a copy of the License at
6-
#
7-
# http://www.apache.org/licenses/LICENSE-2.0
8-
#
9-
# Unless required by applicable law or agreed to in writing, software
10-
# distributed under the License is distributed on an "AS IS" BASIS,
11-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12-
# See the License for the specific language governing permissions and
13-
# limitations under the License.
3+
# Copyright 2018-2022 contributors to the Marquez project
4+
# SPDX-License-Identifier: Apache-2.0
145
#
156
# Usage: $ ./get-docker-compose.sh
167

178
set -e
189

19-
curl -L https://github.com/docker/compose/releases/download/1.25.3/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
20-
chmod +x ~/docker-compose
21-
sudo mv ~/docker-compose /usr/local/bin/docker-compose
22-
docker-compose --version
10+
# Download docker compose
11+
curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
12+
13+
# Change permissions, relocate docker compose, then verify
14+
chmod +x ~/docker-compose && \
15+
sudo mv ~/docker-compose /usr/local/bin/docker-compose && \
16+
docker-compose --version
2317

24-
echo "DONE!"
18+
echo "DONE!"

‎chart/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ Contents of the ```./../docker-compose-postgres..yml``` file can be customized
137137
to better represent your desired setup.
138138

139139
```bash
140-
docker-compose -f ./../docker-compose.postgres.yml -p marquez-postgres up
140+
docker-compose -f ./../docker-compose.db.yml -p marquez-postgres up
141141
```
142142

143143
Once the Postgres instance has been created, run the following command to locate the IP

‎docker-compose.postgres.yml renamed to ‎docker-compose.db.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ version: "3.7"
22
services:
33
db:
44
image: postgres:12.1
5+
container_name: marquez-db
56
ports:
67
- "5432:5432"
78
environment:

‎docker-compose.dev.yml

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,10 @@
11
version: "3.7"
22
services:
33
api:
4-
image: "marquezproject/marquez:${TAG}"
54
build: .
65

7-
web:
8-
build:
9-
context: ./web
10-
dockerfile: Dockerfile
6+
seed_marquez:
7+
build: .
118

129
pghero:
1310
image: ankane/pghero

‎docker-compose.web-dev.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
version: "3.7"
2+
services:
3+
web:
4+
build:
5+
context: ./web
6+
dockerfile: Dockerfile

‎docker-compose.web.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
version: "3.7"
2+
services:
3+
web:
4+
image: "marquezproject/marquez-web:${TAG}"
5+
container_name: marquez-web
6+
environment:
7+
- MARQUEZ_HOST=api
8+
- MARQUEZ_PORT=${API_PORT}
9+
ports:
10+
- "${WEB_PORT}:${WEB_PORT}"
11+
depends_on:
12+
- api

‎docker-compose.yml

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,13 @@ services:
1010
- "${API_PORT}:${API_PORT}"
1111
- "${API_ADMIN_PORT}:${API_ADMIN_PORT}"
1212
volumes:
13-
- utils:/opt/marquez
13+
- data:/opt/marquez
1414
links:
1515
- "db:postgres"
1616
depends_on:
1717
- db
1818
entrypoint: ["/opt/marquez/wait-for-it.sh", "db:5432", "--", "./entrypoint.sh"]
1919

20-
web:
21-
image: "marquezproject/marquez-web:${TAG}"
22-
container_name: marquez-web
23-
environment:
24-
- MARQUEZ_HOST=api
25-
- MARQUEZ_PORT=${API_PORT}
26-
ports:
27-
- "${WEB_PORT}:${WEB_PORT}"
28-
stdin_open: true
29-
tty: true
30-
depends_on:
31-
- api
32-
3320
db:
3421
image: postgres:12.1
3522
container_name: marquez-db
@@ -42,12 +29,15 @@ services:
4229
- MARQUEZ_USER=marquez
4330
- MARQUEZ_PASSWORD=marquez
4431
volumes:
45-
- ./docker/postgresql.conf:/etc/postgresql/postgresql.conf
32+
- db-conf:/etc/postgresql
4633
- db-init:/docker-entrypoint-initdb.d
34+
- db-backup:/var/lib/postgresql/data
4735
command: ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
4836
# Enables SQL statement logging (see: https://www.postgresql.org/docs/12/runtime-config-logging.html#GUC-LOG-STATEMENT)
4937
# command: ["postgres", "-c", "log_statement=all"]
5038

5139
volumes:
52-
utils:
40+
data:
41+
db-conf:
5342
db-init:
43+
db-backup:

‎docker/prune.sh

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,4 @@
77

88
set -e
99

10-
# Change working directory to project root
11-
project_root=$(git rev-parse --show-toplevel)
12-
cd "${project_root}"
13-
1410
docker image prune -a --filter "until=24h" --force

0 commit comments

Comments
 (0)