diff --git a/README.md b/README.md index f6b564b1..c16c1884 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Back up mysql databases to... anywhere! ## Overview -mysql-backup is a simple way to do MySQL database backups and restores. +mysql-backup is a simple way to do MySQL database backups and restores, as well as manage your backups. It has the following features: @@ -14,6 +14,7 @@ It has the following features: * connect to any container running on the same system * select how often to run a dump * select when to start the first dump, whether time of day or relative to container start time +* prune backups older than a specific time period or quantity Please see [CONTRIBUTORS.md](./CONTRIBUTORS.md) for a list of contributors. diff --git a/docs/configuration.md b/docs/configuration.md index 69548c69..d7a4451a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -58,7 +58,7 @@ Various sample configuration files are available in the [sample-configs](../samp ## Configuration Options -The following are the environment variables, CLI flags and configuration file options for a backup or a restore. +The following are the environment variables, CLI flags and configuration file options for: backup(B), restore (R), prune (P). | Purpose | Backup / Restore | CLI Flag | Env Var | Config Key | Default | | --- | --- | --- | --- | --- | --- | @@ -70,21 +70,21 @@ The following are the environment variables, CLI flags and configuration file op | names of databases to exclude from the dump | B | `exclude` | `DB_NAMES_EXCLUDE` | `database.exclude` | | | do not include `USE ;` statement in the dump | B | `no-database-name` | `NO_DATABASE_NAME` | `database.no-database-name` | `false` | | restore to a specific database | R | `restore --database` | `RESTORE_DATABASE` | `restore.database` | | -| how often to do a dump, in minutes | B | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day | -| what time to do the first dump | B | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately | -| cron schedule for dumps | B | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` | | -| run the backup a single time and exit | B | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` | -| enable debug logging | BR | `debug` | `DEBUG` | `logging: debug` | `false` | -| where to put the dump file; see [backup](./backup.md) | B | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` | | +| how often to do a dump or prune, in minutes | BP | `dump --frequency` | `DB_DUMP_FREQ` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day | +| what time to do the first dump or prune | BP | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately | +| cron schedule for dumps or prunes | BP | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` | | +| run the backup or prune a single time and exit | BP | `dump --once` | `RUN_ONCE` | `dump.schedule.once` | `false` | +| enable debug logging | BRP | `debug` | `DEBUG` | `logging: debug` | `false` | +| where to put the dump file; see [backup](./backup.md) | BP | `dump --target` | `DB_DUMP_TARGET` | `dump.targets` | | | where the restore file exists; see [restore](./restore.md) | R | `restore --target` | `DB_RESTORE_TARGET` | `restore.target` | | -| replace any `:` in the dump filename with `-` | B | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` | -| AWS access key ID, used only if a target does not have one | BR | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` | | -| AWS secret access key, used only if a target does not have one | BR | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` | | -| AWS default region, used only if a target does not have one | BR | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` | | +| replace any `:` in the dump filename with `-` | BP | `dump --safechars` | `DB_DUMP_SAFECHARS` | `database.safechars` | `false` | +| AWS access key ID, used only if a target does not have one | BRP | `aws-access-key-id` | `AWS_ACCESS_KEY_ID` | `dump.targets[s3-target].credentials.access-key-id` | | +| AWS secret access key, used only if a target does not have one | BRP | `aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` | `dump.targets[s3-target].credentials.secret-access-key` | | +| AWS default region, used only if a target does not have one | BRP | `aws-region` | `AWS_REGION` | `dump.targets[s3-target].region` | | | alternative endpoint URL for S3-interoperable systems, used only if a target does not have one | BR | `aws-endpoint-url` | `AWS_ENDPOINT_URL` | `dump.targets[s3-target].endpoint` | | -| SMB username, used only if a target does not have one | BR | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` | | -| SMB password, used only if a target does not have one | BR | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` | | -| compression to use, one of: `bzip2`, `gzip` | B | `compression` | `COMPRESSION` | `dump.compression` | `gzip` | +| SMB username, used only if a target does not have one | BRP | `smb-user` | `SMB_USER` | `dump.targets[smb-target].credentials.username` | | +| SMB password, used only if a target does not have one | BRP | `smb-pass` | `SMB_PASS` | `dump.targets[smb-target].credentials.password` | | +| compression to use, one of: `bzip2`, `gzip` | BP | `compression` | `COMPRESSION` | `dump.compression` | `gzip` | | when in container, run the dump or restore with `nice`/`ionice` | BR | `` | `NICE` | `` | `false` | | tmp directory to be used during backup creation and other operations | BR | `tmp` | `TMP_PATH` | `tmp` | system-defined | | filename to save the target backup file | B | `dump --filename-pattern` | `DB_DUMP_FILENAME_PATTERN` | `dump.filename-pattern` | | @@ -92,12 +92,4 @@ The following are the environment variables, CLI flags and configuration file op | directory with scripts to execute after backup | B | `dump --post-backup-scripts` | `DB_DUMP_POST_BACKUP_SCRIPTS` | `dump.scripts.post-backup` | in container, `/scripts.d/post-backup/` | | directory with scripts to execute before restore | R | `restore --pre-restore-scripts` | `DB_DUMP_PRE_RESTORE_SCRIPTS` | `dump.pre-restore-scripts` | in container, `/scripts.d/pre-restore/` | | directory with scripts to execute after restore | R | `restore --post-restore-scripts` | `DB_DUMP_POST_RESTORE_SCRIPTS` | `dump.post-restore-scripts` | in container, `/scripts.d/post-restore/` | - - -## Unsupported Options - -Unsupported options from the old version of `mysql-backup`: - -* `MYSQLDUMP_OPTS`: A string of options to pass to `mysqldump`, e.g. `MYSQLDUMP_OPTS="--opt abc --param def --max_allowed_packet=123455678"` will run `mysqldump --opt abc --param def --max_allowed_packet=123455678`. These are replaced by individual options. -* `AWS_CLI_OPTS`: Additional arguments to be passed to the `aws` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/#options) for a list. These are replaced by target-specific options. -* `AWS_CLI_S3_CP_OPTS`: Additional arguments to be passed to the `s3 cp` part of the `aws s3 cp` command, click [here](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#options) for a list. If you are using AWS KMS, `sse`, `sse-kms-key-id`, etc., may be of interest. These are replaced by target-specific options +| retention policy for backups | BP | `dump --retention` | `RETENTION` | `prune.retention` | Infinite | diff --git a/docs/prune.md b/docs/prune.md new file mode 100644 index 00000000..87c74067 --- /dev/null +++ b/docs/prune.md @@ -0,0 +1,63 @@ +# Pruning + +Pruning is the process of removing backups that no longer are needed. + +mysql-backup does **not** do this by default; it is up to you to enable this feature, if you want it. + +## Launching Pruning + +Pruning happens only in the following scenarios: + +* During pruning runs +* During backup runs + +It does not occur during restore runs. + +### Pruning Runs + +You can start `mysql-backup` with the command `prune` to run a pruning operation. It will prune any backups that are no longer needed. + +Like backups, it can run once and then exit, or it can run on a schedule. + +It uses the same configuration options for scheduling as backups, see the [scheduling](./scheduling.md) documentation for more information, +specifically the section about [Scheduling Options](./scheduling.md#scheduling-options). + +### Backup Runs + +When running `mysql-backup` in backup mode, it _optionally_ can also prune older backups before each backup run. +When enabled, it will prune any backups that fit the pruning criteria. + +## Pruning Criteria + +Pruning can be on the basis of the _age_ of a specific backup, or the _number_ of backups. Both are set by the configuration setting: + +* Environment variable: `RETENTION=` +* CLI flag: `dump --retention=` or `prune --retention=` +* Config file: +```yaml +prune: + retention: +``` + +The value of retention always is an integer followed by a letter. The letter can one of: + +* `h` - hours, e.g. `2h` +* `d` - days, e.g. `3d` +* `w` - weeks, e.g. `4w` +* `m` - months, e.g. `3m` +* `y` - years, e.g. `5y` +* `c` - count, how many backup to keep, e.g. `10c`; this could have been simply `10`, but was kept as `c` to avoid accidental confusion with the other options. + +Most of these are interchangeable, e.g. `3d` is the same as `72h`, and `4w` is the same as `28d` is the same as `672h`. + +When calculating whether or not to prune, `mysql-backup` __always__ converts the amount to hours, and then errs on the side of caution. +For example, if provided `7d`, it will convert that to `168h`, and then prune any backups older than 168 full hours. If it is 167 hours and 59 minutes old, it +will not be pruned. + +## Determining backup age + +Pruning depends on the name of the backup file, rather than the timestamp on the target filesystem, as the latter can be unreliable. +This means that the filename must be of a known pattern. + +As of this writing, pruning only work for backup files whose filename uses the default naming scheme, as described in +["Dump File" in backup documentation](./backup.md#dump-file). We hope to support custom filenames in the future.