[Request] Borgmatic to replace zip method as core backup function

[EkoNesLeg]

Integrating Borgmatic into UNIT3D as the core backup function would provide a much needed upgrade over the existing zip-based method. Borg has features like snapshots, deduplication, and incremental backups, which conserves resources and significantly reduces storage space by only saving changes since the last backup, rather than duplicating entire datasets. It also has native encryption. For example, a 14GB site can maintain dozens of hourly, nightly, and monthly snapshots and restoration points for both site and database, using only a fraction of the space.

borgmatic rinfo

Archive fingerprint: dc03671cfdafasdfa8fasdfkla39423420239f56b45eb969cabb9c3b36f8ce88708
Comment:
Hostname: unit3d.local
Username: ubuntu
Time (start): Mon, 2024-04-08 12:00:05
Time (end): Mon, 2024-04-08 12:00:46
Duration: 21.52 seconds
Number of files: 143858
Command line: /usr/bin/borg create --exclude-from /tmp/tmp1x13ybwl --one-file-system --read-special --list --filter AME- --info --stats '/home/ubuntu/backups/unit3d.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /home/ubuntu/.borgmatic /home/ubuntu/.config/borgmatic/config.yaml /var/www/html
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               14.21 GB             10.69 GB            686.97 MB
All archives:              819.80 GB            617.86 GB             37.34 GB
                       Unique chunks         Total chunks
Chunk index:                  183684              8636948

Objective

The goal is to replace UNIT3D's existing backup system with a Borg-based one. This will introduce frontend functions for initiating backups, extracting/restoring files or directories, and performing database restorations, with configuration management and cronjobs handled in the backend.

Pre-requisites

• Borgmatic installed on the UNIT3D server.
• A Borgmatic configuration file (~/.config/borgmatic/config.yaml). Here's a working example being used for UNIT3D:

encryption_passphrase: ${BORG_PASSPHRASE}

source_directories:
  - /var/www/html

repositories:
  - path: ssh://[email protected]:22/backup/unit3d.borg
    label: offsite
  - path: /var/www/html/storage/backups/unit3d.borg
    label: local

keep_hourly: 24
keep_daily: 30
keep_weekly: 12
keep_monthly: 12
keep_yearly: 7

checks:
  - name: repository
  - name: archives
    frequency: 2 weeks

exclude_patterns:
  - /var/www/html/storage/backups
  - /var/www/html/storage/app/backup-temp/

mysql_databases:
  - name: unit3d
    username: root
    password: ${MYSQL_ROOT_PASSWORD}
    format: sql

• Initiate the Borg repository on the server with the following command:

borgmatic rcreate --encryption repokey-aes-ocb

• Cronjobs configured for running borgmatic at specific intervals.

Suggested Buttons for UNIT3D Sysop Dashboard

• Schedule Backups - For setting up and modifying backup cronjob schedules.

crontab -e

• Backup Now - To initiate an immediate backup.

borgmatic --verbosity 1 --list --stats >> /var/www/html/storage/logs/unit3d_borgmatic/backup_now.log 2>&1

• View Backup Logs - For accessing detailed logs of each backup job.

cat /var/www/html/storage/logs/four_times_daily.log

• List Archives - To view available backup archives.

borgmatic rlist

• Restore Latest Snapshot - For extracting the most recent backup.

borgmatic extract --archive latest

• Restore Particular Snapshot - To extract a specific backup archive.

borgmatic extract --archive host-2023-01-02T04:06:07.080910

• Restore Torrent Files - For restoring only the torrent files from the latest backup.

borgmatic extract --archive latest --path /var/www/html/public/files/torrents

• Restore Latest Database Snapshot - To restore the most recent database backup.

borgmatic restore --archive latest

• Restore Particular Database Snapshot - For restoring a database from a specific backup.

borgmatic restore --archive host-2023-01-02T04:06:07.080910

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.