sqlite: Optimize the DB and the files sizes #4602
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It should have been done in the migration of version 7, to reduce the size of the database on the filesystem after the media cache was moved to the SqliteEventCacheStore. Better late than never. Signed-off-by: Kévin Commaille <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4602 +/- ##
=======================================
Coverage 85.73% 85.73%
=======================================
Files 291 291
Lines 33288 33303 +15
=======================================
+ Hits 28538 28552 +14
- Misses 4750 4751 +1 ☔ View full report in Codecov by Sentry. |
As recommended by the SQLite docs. Signed-off-by: Kévin Commaille <[email protected]>
zecakeh
force-pushed
the
optimize-sqlite
branch
2 times, most recently
from
2e091d8 to
8222a7d
Compare
The WAL file can grow depending on the transactions that are run. A critical case is VACUUM which basically writes the content of the DB file to the WAL file before writing it back to the DB file. SQLite doesn't try to reduce the size of the file after that unless we set an explicit limit, so we could end up taking twice the size of the database on the filesystem. Signed-off-by: Kévin Commaille <[email protected]>
Signed-off-by: Kévin Commaille <[email protected]>
zecakeh
force-pushed
the
optimize-sqlite
branch
from
8222a7d to
c857daf
Compare
poljar
approved these changes
Feb 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
VACUUMon theSqliteStateStore. We should have done this when we moved the media cache toSqlitEventCacheStore, to defragment the file and reduce its size on the filesystem. Because of that there are databases from that time that still take much more space than they need.PRAGMA optimizeafter running the eventual migrations. The SQLite docs recommend to do this regularly, especially after a schema change. Doing this consistently during construction seems to be the easiest solution.VACUUMwhich basically writes the content of the DB file to the WAL file before writing it back to the DB file, so we end up taking twice the size of the database. The docs say that the WAL file is written to the database when it reaches 4MB, so 10MB seems like a good limit that shouldn't truncate the file too often.