Skip to content

Add snappy streaming compression to WAL segments#70

Merged
nakabonne merged 1 commit into
mainfrom
emdash/feat-compress-wal-file-62d
Mar 16, 2026
Merged

Add snappy streaming compression to WAL segments#70
nakabonne merged 1 commit into
mainfrom
emdash/feat-compress-wal-file-62d

Conversation

@nakabonne

Copy link
Copy Markdown
Owner

Close #47

Why

WAL files grow rapidly under load . The root cause is that metric names are written uncompressed and repeated on every row, making WAL data highly compressible.

What

Introduce Snappy streaming compression to WAL segment files, targeting 2-3x file size reduction.

Changes:

  • Writer chain: Insert snappy.Writer between bufio.Writer and os.File (bufio → snappy → file)
  • Reader chain: Wrap segment file reads with snappy.NewReader for transparent decompression
  • Helper methods: Extract initWriter / closeWriter to consolidate writer chain setup and teardown across
    newDiskWAL, punctuate, refresh, and removeAll
  • Crash recovery: Add snappy.ErrCorrupt tolerance in readAll() — Snappy's frame format is self-synchronizing, so
    partial frames from a crash are safely skipped
  • Test: Verify segment files contain the Snappy stream magic bytes (\xff\x06\x00\x00sNaPpY)

Backward compatibility

WAL segments are deleted on clean shutdown, so no migration is needed. The only edge case is a crash during
upgrade, where old uncompressed segments would fail to read — this is acceptable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 50.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.44%. Comparing base (64a7605) to head (168b642).

Files with missing lines Patch % Lines
disk_wal.go 50.00% 7 Missing and 5 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #70      +/-   ##
==========================================
- Coverage   60.45%   60.44%   -0.01%     
==========================================
  Files          19       19              
  Lines        1613     1625      +12     
==========================================
+ Hits          975      982       +7     
- Misses        502      505       +3     
- Partials      136      138       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nakabonne nakabonne merged commit cec3a95 into main Mar 16, 2026
4 checks passed
@nakabonne nakabonne deleted the emdash/feat-compress-wal-file-62d branch March 16, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compress WAL file

2 participants