Skip to content

Commit 364e0f8

Browse files
jkanindajkaninda
authored
feat: add backup all databases (#171)
1 parent 5b079f7 commit 364e0f8

File tree

20 files changed

+796
-220
lines changed

20 files changed

+796
-220
lines changed

‎.github/workflows/tests.yml

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- nightly
8+
- develop
9+
pull_request:
10+
branches:
11+
- main
12+
env:
13+
IMAGE_NAME: pg-bkup
14+
DB_USERNAME: postgres
15+
DB_PASSWORD: password
16+
17+
jobs:
18+
tests:
19+
runs-on: ubuntu-latest
20+
services:
21+
postgres:
22+
image: postgres:17
23+
env:
24+
POSTGRES_USER: ${{ env.DB_USERNAME }}
25+
POSTGRES_PASSWORD: ${{ env.DB_PASSWORD }}
26+
POSTGRES_DB: testdb
27+
ports:
28+
- 5432:5432
29+
options: >-
30+
--health-cmd="pg_isready"
31+
--health-interval=10s
32+
--health-timeout=5s
33+
--health-retries=5
34+
postgres15:
35+
image: postgres:15
36+
env:
37+
POSTGRES_USER: ${{ env.DB_USERNAME }}
38+
POSTGRES_PASSWORD: ${{ env.DB_PASSWORD }}
39+
POSTGRES_DB: testdb
40+
ports:
41+
- 5433:5432
42+
options: >-
43+
--health-cmd="pg_isready"
44+
--health-interval=10s
45+
--health-timeout=5s
46+
--health-retries=5
47+
postgres10:
48+
image: postgres:10
49+
env:
50+
POSTGRES_USER: ${{ env.DB_USERNAME }}
51+
POSTGRES_PASSWORD: ${{ env.DB_PASSWORD }}
52+
POSTGRES_DB: testdb
53+
ports:
54+
- 5434:5432
55+
options: >-
56+
--health-cmd="pg_isready"
57+
--health-interval=10s
58+
--health-timeout=5s
59+
--health-retries=5
60+
steps:
61+
- name: Checkout repository
62+
uses: actions/checkout@v4
63+
64+
- name: Set up Docker Buildx
65+
uses: docker/setup-buildx-action@v3
66+
- name: Create Minio container
67+
run: |
68+
docker run -d --rm --name minio \
69+
--network host \
70+
-p 9000:9000 \
71+
-e MINIO_ACCESS_KEY=minioadmin \
72+
-e MINIO_SECRET_KEY=minioadmin \
73+
-e MINIO_REGION_NAME="eu" \
74+
minio/minio server /data
75+
echo "Create Minio container completed"
76+
- name: Install PostgreSQL client
77+
run: |
78+
sudo apt-get update
79+
sudo apt-get install -y postgresql-client
80+
- name: Create additional databases
81+
run: |
82+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5432 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb2;"
83+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5432 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb3;"
84+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5432 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE fakedb;"
85+
86+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5433 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb2;"
87+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5433 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb3;"
88+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5433 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE fakedb;"
89+
90+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5434 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb2;"
91+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5434 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE testdb3;"
92+
PGPASSWORD=${{ env.DB_PASSWORD }} psql -h localhost -p 5434 -U ${{ env.DB_USERNAME }} -c "CREATE DATABASE fakedb;"
93+
- name: Install MinIO Client (mc)
94+
run: |
95+
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
96+
chmod +x mc
97+
sudo mv mc /usr/local/bin/
98+
99+
- name: Wait for MinIO to be ready
100+
run: sleep 5
101+
102+
- name: Configure MinIO Client
103+
run: |
104+
mc alias set local http://localhost:9000 minioadmin minioadmin
105+
mc alias list
106+
107+
- name: Create MinIO Bucket
108+
run: |
109+
mc mb local/backups
110+
echo "Bucket backups created successfully."
111+
# Build the Docker image
112+
- name: Build Docker Image
113+
run: |
114+
docker buildx build --build-arg appVersion=test -t ${{ env.IMAGE_NAME }}:latest --load .
115+
116+
- name: Verify Docker images
117+
run: |
118+
docker images
119+
120+
- name: Test restore
121+
run: |
122+
docker run --rm --name ${{ env.IMAGE_NAME }} \
123+
-v ./migrations:/backup/ \
124+
--network host \
125+
-e DB_HOST=127.0.0.1 \
126+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
127+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
128+
-e DB_NAME=testdb \
129+
${{ env.IMAGE_NAME }}:latest restore -f init.sql
130+
echo "Database restore completed"
131+
- name: Test restore Postgres15
132+
run: |
133+
docker run --rm --name ${{ env.IMAGE_NAME }} \
134+
-v ./migrations:/backup/ \
135+
--network host \
136+
-e DB_HOST=127.0.0.1 \
137+
-e DB_PORT=5433 \
138+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
139+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
140+
-e DB_NAME=testdb \
141+
${{ env.IMAGE_NAME }}:latest restore -f init.sql
142+
echo "Test restore Postgres15 completed"
143+
- name: Test restore Postgres10
144+
run: |
145+
docker run --rm --name ${{ env.IMAGE_NAME }} \
146+
-v ./migrations:/backup/ \
147+
--network host \
148+
-e DB_HOST=127.0.0.1 \
149+
-e DB_PORT=5434 \
150+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
151+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
152+
-e DB_NAME=testdb \
153+
${{ env.IMAGE_NAME }}:latest restore -f init.sql
154+
echo "Test restore Postgres10 completed"
155+
- name: Test backup
156+
run: |
157+
docker run --rm --name ${{ env.IMAGE_NAME }} \
158+
-v ./migrations:/backup/ \
159+
--network host \
160+
-e DB_HOST=127.0.0.1 \
161+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
162+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
163+
-e DB_NAME=testdb \
164+
${{ env.IMAGE_NAME }}:latest backup
165+
echo "Database backup completed"
166+
- name: Test backup Postgres15
167+
run: |
168+
docker run --rm --name ${{ env.IMAGE_NAME }} \
169+
-v ./migrations:/backup/ \
170+
--network host \
171+
-e DB_PORT=5433 \
172+
-e DB_HOST=127.0.0.1 \
173+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
174+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
175+
-e DB_NAME=testdb \
176+
${{ env.IMAGE_NAME }}:latest backup
177+
echo "Test backup Postgres15 completed"
178+
- name: Test backup Postgres10
179+
run: |
180+
docker run --rm --name ${{ env.IMAGE_NAME }} \
181+
-v ./migrations:/backup/ \
182+
--network host \
183+
-e DB_PORT=5434 \
184+
-e DB_HOST=127.0.0.1 \
185+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
186+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
187+
-e DB_NAME=testdb \
188+
${{ env.IMAGE_NAME }}:latest backup
189+
echo "Test backup Postgres10 completed"
190+
- name: Test encrypted backup
191+
run: |
192+
docker run --rm --name ${{ env.IMAGE_NAME }} \
193+
-v ./migrations:/backup/ \
194+
--network host \
195+
-e DB_HOST=127.0.0.1 \
196+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
197+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
198+
-e GPG_PASSPHRASE=password \
199+
${{ env.IMAGE_NAME }}:latest backup -d testdb --disable-compression --custom-name encrypted-bkup
200+
echo "Database encrypted backup completed"
201+
- name: Test restore encrypted backup | testdb -> testdb2
202+
run: |
203+
docker run --rm --name ${{ env.IMAGE_NAME }} \
204+
-v ./migrations:/backup/ \
205+
--network host \
206+
-e DB_HOST=127.0.0.1 \
207+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
208+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
209+
-e GPG_PASSPHRASE=password \
210+
-e DB_NAME=testdb2 \
211+
${{ env.IMAGE_NAME }}:latest restore -f /backup/encrypted-bkup.sql.gpg
212+
echo "Test restore encrypted backup completed"
213+
- name: Test migrate database testdb -> testdb3
214+
run: |
215+
docker run --rm --name ${{ env.IMAGE_NAME }} \
216+
-v ./migrations:/backup/ \
217+
--network host \
218+
-e DB_HOST=127.0.0.1 \
219+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
220+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
221+
-e GPG_PASSPHRASE=password \
222+
-e DB_NAME=testdb \
223+
-e TARGET_DB_HOST=127.0.0.1 \
224+
-e TARGET_DB_PORT=5432 \
225+
-e TARGET_DB_NAME=testdb3 \
226+
-e TARGET_DB_USERNAME=${{ env.DB_USERNAME }} \
227+
-e TARGET_DB_PASSWORD=${{ env.DB_PASSWORD }} \
228+
${{ env.IMAGE_NAME }}:latest migrate
229+
echo "Test migrate database testdb -> testdb3 completed"
230+
- name: Test backup all databases
231+
run: |
232+
docker run --rm --name ${{ env.IMAGE_NAME }} \
233+
-v ./migrations:/backup/ \
234+
--network host \
235+
-e DB_HOST=127.0.0.1 \
236+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
237+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
238+
-e DB_NAME=testdb \
239+
${{ env.IMAGE_NAME }}:latest backup --all-databases
240+
echo "Database backup completed"
241+
- name: Test multiple backup
242+
run: |
243+
docker run --rm --name ${{ env.IMAGE_NAME }} \
244+
-v ./migrations:/backup/ \
245+
--network host \
246+
-e DB_HOST=127.0.0.1 \
247+
-e TESTDB_DB_USERNAME=${{ env.DB_USERNAME }} \
248+
-e TESTDB_DB_PASSWORD=${{ env.DB_PASSWORD }} \
249+
-e TESTDB2_DB_USERNAME=${{ env.DB_USERNAME }} \
250+
-e TESTDB2_DB_PASSWORD=${{ env.DB_PASSWORD }} \
251+
-e TESTDB2_DB_HOST=127.0.0.1 \
252+
${{ env.IMAGE_NAME }}:latest backup -c /backup/test_config.yaml
253+
echo "Database backup completed"
254+
- name: Test backup Minio (s3)
255+
run: |
256+
docker run --rm --name ${{ env.IMAGE_NAME }} \
257+
--network host \
258+
-e DB_HOST=127.0.0.1 \
259+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
260+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
261+
-e DB_NAME=testdb \
262+
-e AWS_S3_ENDPOINT="http://127.0.0.1:9000" \
263+
-e AWS_S3_BUCKET_NAME=backups \
264+
-e AWS_ACCESS_KEY=minioadmin \
265+
-e AWS_SECRET_KEY=minioadmin \
266+
-e AWS_DISABLE_SSL="true" \
267+
-e AWS_REGION="eu" \
268+
-e AWS_FORCE_PATH_STYLE="true" ${{ env.IMAGE_NAME }}:latest backup -s s3 --custom-name minio-backup
269+
echo "Test backup Minio (s3) completed"
270+
- name: Test restore Minio (s3)
271+
run: |
272+
docker run --rm --name ${{ env.IMAGE_NAME }} \
273+
--network host \
274+
-e DB_HOST=127.0.0.1 \
275+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
276+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
277+
-e DB_NAME=testdb \
278+
-e AWS_S3_ENDPOINT="http://127.0.0.1:9000" \
279+
-e AWS_S3_BUCKET_NAME=backups \
280+
-e AWS_ACCESS_KEY=minioadmin \
281+
-e AWS_SECRET_KEY=minioadmin \
282+
-e AWS_DISABLE_SSL="true" \
283+
-e AWS_REGION="eu" \
284+
-e AWS_FORCE_PATH_STYLE="true" ${{ env.IMAGE_NAME }}:latest restore -s s3 -f minio-backup.sql.gz
285+
echo "Test backup Minio (s3) completed"
286+
- name: Test scheduled backup
287+
run: |
288+
docker run -d --rm --name ${{ env.IMAGE_NAME }} \
289+
-v ./migrations:/backup/ \
290+
--network host \
291+
-e DB_HOST=127.0.0.1 \
292+
-e DB_USERNAME=${{ env.DB_USERNAME }} \
293+
-e DB_PASSWORD=${{ env.DB_PASSWORD }} \
294+
-e DB_NAME=testdb \
295+
${{ env.IMAGE_NAME }}:latest backup -e "@every 10s"
296+
297+
echo "Waiting for backup to be done..."
298+
sleep 25
299+
docker logs ${{ env.IMAGE_NAME }}
300+
echo "Test scheduled backup completed"
301+
# Cleanup: Stop and remove containers
302+
- name: Clean up
303+
run: |
304+
docker stop ${{ env.IMAGE_NAME }} || true
305+
docker rm ${{ env.IMAGE_NAME }} || true

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
**PG-BKUP** is a Docker container image designed to **backup, restore, and migrate PostgreSQL databases**.
44
It supports a variety of storage options and ensures data security through GPG encryption.
55

6+
[![Tests](https://github.com/jkaninda/pg-bkup/actions/workflows/tests.yml/badge.svg)](https://github.com/jkaninda/pg-bkup/actions/workflows/tests.yml)
67
[![Build](https://github.com/jkaninda/pg-bkup/actions/workflows/release.yml/badge.svg)](https://github.com/jkaninda/pg-bkup/actions/workflows/release.yml)
78
[![Go Report](https://goreportcard.com/badge/github.com/jkaninda/mysql-bkup)](https://goreportcard.com/report/github.com/jkaninda/pg-bkup)
89
![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/jkaninda/pg-bkup?style=flat-square)

‎cmd/backup.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,5 +51,7 @@ func init() {
5151
BackupCmd.PersistentFlags().StringP("cron-expression", "e", "", "Backup cron expression (e.g., `0 0 * * *` or `@daily`)")
5252
BackupCmd.PersistentFlags().StringP("config", "c", "", "Configuration file for multi database backup. (e.g: `/backup/config.yaml`)")
5353
BackupCmd.PersistentFlags().BoolP("disable-compression", "", false, "Disable backup compression")
54-
54+
BackupCmd.PersistentFlags().BoolP("all-databases", "a", false, "Backup all databases")
55+
BackupCmd.PersistentFlags().BoolP("all-in-one", "A", false, "Backup all databases in a single file")
56+
BackupCmd.PersistentFlags().StringP("custom-name", "", "", "Custom backup name")
5557
}

‎docs/how-tos/backup-all.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
title: Backup all databases in the server
3+
layout: default
4+
parent: How Tos
5+
nav_order: 12
6+
---
7+
8+
# Backup All Databases
9+
10+
PG-Bkup supports backing up all databases on the server using the `--all-databases` (`-a`) flag. By default, this creates separate backup files for each database. If you prefer a single backup file, you can use the `--all-in-one` (`-A`) flag.
11+
12+
Backing up all databases is useful for creating a snapshot of the entire database server, whether for disaster recovery or migration purposes.
13+
## Backup Modes
14+
15+
### Separate Backup Files (Default)
16+
17+
Using --all-databases without --all-in-one creates individual backup files for each database.
18+
19+
- Creates separate backup files for each database.
20+
- Provides more flexibility in restoring individual databases or tables.
21+
- Can be more manageable in cases where different databases have different retention policies.
22+
- Might take slightly longer due to multiple file operations.
23+
- It is the default behavior when using the `--all-databases` flag.
24+
- It does not backup system databases (`postgres`,`template0`, `template1`,...).
25+
26+
**Command:**
27+
28+
```bash
29+
docker run --rm --network your_network_name \
30+
-v $PWD/backup:/backup/ \
31+
-e "DB_HOST=dbhost" \
32+
-e "DB_PORT=5432" \
33+
-e "DB_USERNAME=username" \
34+
-e "DB_PASSWORD=password" \
35+
jkaninda/pg-bkup backup --all-databases
36+
```
37+
### Single Backup File
38+
39+
Using --all-in-one (-A) creates a single backup file containing all databases.
40+
41+
- Creates a single backup file containing all databases.
42+
- Easier to manage if you need to restore everything at once.
43+
- Faster to back up and restore in bulk.
44+
- Can be problematic if you only need to restore a specific database or table.
45+
- It is recommended to use this option for disaster recovery purposes.
46+
- It backups system databases as well.
47+
48+
```bash
49+
docker run --rm --network your_network_name \
50+
-v $PWD/backup:/backup/ \
51+
-e "DB_HOST=dbhost" \
52+
-e "DB_PORT=5432" \
53+
-e "DB_USERNAME=username" \
54+
-e "DB_PASSWORD=password" \
55+
jkaninda/pg-bkup backup --all-in-one
56+
```
57+
58+
### When to Use Which?
59+
60+
- Use `--all-in-one` if you want a quick, simple backup for disaster recovery where you'll restore everything at once.
61+
- Use `--all-databases` if you need granularity in restoring specific databases or tables without affecting others.

‎docs/how-tos/receive-notification.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: Receive notifications
33
layout: default
44
parent: How Tos
5-
nav_order: 12
5+
nav_order: 13
66
---
77

88
# Receive Notifications

0 commit comments

Comments
 (0)