files cache with size #7846
files cache with size #7846
Conversation
|
This is trying to prepare a bigger change in how borg works: giving up with precise refcounting and removing chunk size information from the chunks index/cache (so borg could just work with the set of existing chunks with no further information about them in the chunks index/cache). A first step into that direction is adding the chunk sizes to the files cache (it currently only has the chunk ids). If borg create detects a file is unchanged, we need to have that information at hand to create the item's chunk list without reading/chunking the file. NewCache tries to be similar to AdHocCache, but additionally has the files cache. |
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
3 times, most recently
from
527b612
to
aef7d7a
Compare
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #7846 +/- ##
==========================================
- Coverage 83.75% 82.83% -0.93%
==========================================
Files 67 67
Lines 12061 12166 +105
Branches 2189 2197 +8
==========================================
- Hits 10102 10078 -24
- Misses 1367 1491 +124
- Partials 592 597 +5 ☔ View full report in Codecov by Sentry. |
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
5 times, most recently
from
cdf705b
to
f404810
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
7490f37
to
c5e5ce6
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
c5e5ce6
to
5c9dcc0
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
5c9dcc0
to
02184e0
Compare
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
02184e0
to
dc25787
Compare
the files cache used to have only the chunk ids, so it had to rely on the chunks index having the size information - which is problematic with e.g. the AdhocCache (has size==0 for all not new chunks) and blocked using the files cache there.
incref: returns (id, size), so it needs the size if it can't get it from the chunks index. also needed for updating stats. decref: caller does not always have the chunk size (e.g. for metadata chunks), as we consider 0 to be an invalid size, we call with size == 1 in that case. thus, stats might be slightly off.
when given, force using the AdHocCache.
if a chunk is missing in repo, it will also be missing in a ad-hoc built chunks index.
thus: - no cache.path - skip on-disk cache corruption tests for AdHocCache
Also: - move common code to ChunksMixin - always use ._txn_active (not .txn_active)
if we use AdHocCache or NewCache, we do not have precise refcounting. thus, we do not delete repo objects as their refcount does not go to zero. check --repair will just remove the orphans.
Only LocalCache implements these.
ThomasWaldmann
force-pushed
the
files-cache-with-size
branch
from
dc25787
to
1389bd1
Compare
No description provided.