bottomless-cli: added snapshot command #1229
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends bottomless-cli with an ability to generate a snapshot option. Example usage:
If you don't know what generations are out there you can run:
To test - given that you have a local bottomless storage with at least 2 generations dependent on each other:
.dep
file. Removedb.zstd
and.changecounter
files - this will make bottomless treat the generation as if it had no snapshots.export RUST_LOG=info,bottomless=debug
for better visibility. At this point bottomless still can restore the database eg.bottomless-cli -e "http://localhost:9000/" -b "bottomless" -d "./data.sqld" -n "ns-<db-id>:<namespace>" restore -g "<generation>"
, however it will use.dep
file to reach back across many generations to restore the db. At the end of restore a SQLite integrity check is being done.bottomless-cli -e "http://localhost:9000/" -b "bottomless" -d "./data.sqld" -n "ns-<db-id>:<namespace>" snapshot -g "<generation>"
. It will recreate and re-upload a snapshot at the beginning of a generation. If you'll try to restore db again you'll see that a regenerated snapshot is being used, so that restore process is faster.The idea here is that this way we could skip creating snapshots from within the database VMs themselves, and instead have a separate machine/cron job that creates them periodically in the background without putting the pressure related to uploading big DB snapshots from database machine.
Additionally, snapshot creation also performs a local restore and integrity check before upload, so we also can confirm that we can restore database to an uncorrupted state from existing backup.