Skip to content

Commit

Permalink
enable retention
Browse files Browse the repository at this point in the history
Signed-off-by: Avi Deitcher <[email protected]>
  • Loading branch information
deitch committed Mar 12, 2024
1 parent 738a5d6 commit 5df15dd
Show file tree
Hide file tree
Showing 3 changed files with 80 additions and 24 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Back up mysql databases to... anywhere!

## Overview

mysql-backup is a simple way to do MySQL database backups and restores.
mysql-backup is a simple way to do MySQL database backups and restores, as well as manage your backups.

It has the following features:

Expand All @@ -14,6 +14,7 @@ It has the following features:
* connect to any container running on the same system
* select how often to run a dump
* select when to start the first dump, whether time of day or relative to container start time
* prune backups older than a specific time period or quantity

Please see [CONTRIBUTORS.md](./CONTRIBUTORS.md) for a list of contributors.

Expand Down
38 changes: 15 additions & 23 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Various sample configuration files are available in the [sample-configs](../samp

## Configuration Options

The following are the environment variables, CLI flags and configuration file options for a backup or a restore.
The following are the environment variables, CLI flags and configuration file options for: backup(B), restore (R), prune (P).

| Purpose | Backup / Restore | CLI Flag | Env Var | Config Key | Default |
| --- | --- | --- | --- | --- | --- |
Expand All @@ -70,34 +70,26 @@ The following are the environment variables, CLI flags and configuration file op
| names of databases to exclude from the dump | B | `exclude` | `DB_NAMES_EXCLUDE` | `database.exclude` | |
| do not include `USE <database>;` statement in the dump | B | `no-database-name` | `NO_DATABASE_NAME` | `database.no-database-name` | `false` |
| restore to a specific database | R | `restore --database` | `RESTORE_DATABASE` | `restore.database` | |
| how often to do a dump, in minutes | B | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day |
| what time to do the first dump | B | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately |
| cron schedule for dumps | B | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` | |
| run the backup a single time and exit | B | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` |
| enable debug logging | BR | `debug` | `DEBUG` | `logging: debug` | `false` |
| where to put the dump file; see [backup](./backup.md) | B | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` | |
| how often to do a dump or prune, in minutes | BP | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day |
| what time to do the first dump or prune | BP | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately |
| cron schedule for dumps or prunes | BP | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` | |
| run the backup or prune a single time and exit | BP | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` |
| enable debug logging | BRP | `debug` | `DEBUG` | `logging: debug` | `false` |
| where to put the dump file; see [backup](./backup.md) | BP | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` | |
| where the restore file exists; see [restore](./restore.md) | R | `restore --target` | `DB_RESTORE_TARGET` | `restore.target` | |
| replace any `:` in the dump filename with `-` | B | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` |
| AWS access key ID, used only if a target does not have one | BR | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` | |
| AWS secret access key, used only if a target does not have one | BR | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` | |
| AWS default region, used only if a target does not have one | BR | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` | |
| replace any `:` in the dump filename with `-` | BP | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` |
| AWS access key ID, used only if a target does not have one | BRP | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` | |
| AWS secret access key, used only if a target does not have one | BRP | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` | |
| AWS default region, used only if a target does not have one | BRP | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` | |
| alternative endpoint URL for S3-interoperable systems, used only if a target does not have one | BR | `aws-endpoint-url` | `AWS_ENDPOINT_URL` | `dump.targets[s3-target].endpoint` | |
| SMB username, used only if a target does not have one | BR | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` | |
| SMB password, used only if a target does not have one | BR | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` | |
| compression to use, one of: `bzip2`, `gzip` | B | `compression` | `COMPRESSION` | `dump.compression` | `gzip` |
| SMB username, used only if a target does not have one | BRP | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` | |
| SMB password, used only if a target does not have one | BRP | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` | |
| compression to use, one of: `bzip2`, `gzip` | BP | `compression` | `COMPRESSION` | `dump.compression` | `gzip` |
| when in container, run the dump or restore with `nice`/`ionice` | BR | `` | `NICE` | `` | `false` |
| tmp directory to be used during backup creation and other operations | BR | `tmp` | `TMP_PATH` | `tmp` | system-defined |
| filename to save the target backup file | B | `dump --filename-pattern` | `DB_DUMP_FILENAME_PATTERN` | `dump.filename-pattern` | |
| directory with scripts to execute before backup | B | `dump --pre-backup-scripts` | `DB_DUMP_PRE_BACKUP_SCRIPTS` | `dump.scripts.pre-backup` | in container, `/scripts.d/pre-backup/` |
| directory with scripts to execute after backup | B | `dump --post-backup-scripts` | `DB_DUMP_POST_BACKUP_SCRIPTS` | `dump.scripts.post-backup` | in container, `/scripts.d/post-backup/` |
| directory with scripts to execute before restore | R | `restore --pre-restore-scripts` | `DB_DUMP_PRE_RESTORE_SCRIPTS` | `dump.pre-restore-scripts` | in container, `/scripts.d/pre-restore/` |
| directory with scripts to execute after restore | R | `restore --post-restore-scripts` | `DB_DUMP_POST_RESTORE_SCRIPTS` | `dump.post-restore-scripts` | in container, `/scripts.d/post-restore/` |


## Unsupported Options

Unsupported options from the old version of `mysql-backup`:

* `MYSQLDUMP_OPTS`: A string of options to pass to `mysqldump`, e.g. `MYSQLDUMP_OPTS="--opt abc --param def --max_allowed_packet=123455678"` will run `mysqldump --opt abc --param def --max_allowed_packet=123455678`. These are replaced by individual options.
* `AWS_CLI_OPTS`: Additional arguments to be passed to the `aws` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/#options) for a list. These are replaced by target-specific options.
* `AWS_CLI_S3_CP_OPTS`: Additional arguments to be passed to the `s3 cp` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#options) for a list. If you are using AWS KMS, `sse`, `sse-kms-key-id`, etc., may be of interest. These are replaced by target-specific options
| retention policy for backups | BP | `dump --retention` | `RETENTION` | `prune.retention` | Infinite |
63 changes: 63 additions & 0 deletions docs/prune.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Pruning

Pruning is the process of removing backups that no longer are needed.

mysql-backup does **not** do this by default; it is up to you to enable this feature, if you want it.

## Launching Pruning

Pruning happens only in the following scenarios:

* During pruning runs
* During backup runs

It does not occur during restore runs.

### Pruning Runs

You can start `mysql-backup` with the command `prune` to run a pruning operation. It will prune any backups that are no longer needed.

Like backups, it can run once and then exit, or it can run on a schedule.

It uses the same configuration options for scheduling as backups, see the [scheduling](./scheduling.md) documentation for more information,
specifically the section about [Scheduling Options](./scheduling.md#scheduling-options).

### Backup Runs

When running `mysql-backup` in backup mode, it _optionally_ can also prune older backups before each backup run.
When enabled, it will prune any backups that fit the pruning criteria.

## Pruning Criteria

Pruning can be on the basis of the _age_ of a specific backup, or the _number_ of backups. Both are set by the configuration setting:

* Environment variable: `RETENTION=<value>`
* CLI flag: `dump --retention=<value>` or `prune --retention=<value>`
* Config file:
```yaml
prune:
retention: <value>
```
The value of retention always is an integer followed by a letter. The letter can one of:
* `h` - hours, e.g. `2h`
* `d` - days, e.g. `3d`
* `w` - weeks, e.g. `4w`
* `m` - months, e.g. `3m`
* `y` - years, e.g. `5y`
* `c` - count, how many backup to keep, e.g. `10c`; this could have been simply `10`, but was kept as `c` to avoid accidental confusion with the other options.

Most of these are interchangeable, e.g. `3d` is the same as `72h`, and `4w` is the same as `28d` is the same as `672h`.

When calculating whether or not to prune, `mysql-backup` __always__ converts the amount to hours, and then errs on the side of caution.
For example, if provided `7d`, it will convert that to `168h`, and then prune any backups older than 168 full hours. If it is 167 hours and 59 minutes old, it
will not be pruned.

## Determining backup age

Pruning depends on the name of the backup file, rather than the timestamp on the target filesystem, as the latter can be unreliable.
This means that the filename must be of a known pattern.

As of this writing, pruning only work for backup files whose filename uses the default naming scheme, as described in
["Dump File" in backup documentation](./backup.md#dump-file). We hope to support custom filenames in the future.

0 comments on commit 5df15dd

Please sign in to comment.