Compressed EROFS from mkcomposefs?

[Winterhuman]

#332 (comment) mentions: "The erofs produced by mkcomposefs on a reasonably complete /usr is on the order of double digits MB. I've seen ~50MB generally, and it compresses well (down to more like 10MB)."

However, I can't find anything on how to get a compressed EROFS image from mkcomposefs. Is it already the default to compress it (in which case, where does 50MB come from?), or was the 10MB image made by converting the original EROFS image somehow (and if so, how?).

[cgwalters]

EROFS doesn't compress metadata, only data. The composefs EROFS is just metadata.

There's nothing stopping one from compressing it out of band and decompressing it before use, although that would currently mean the fsverity tree would need to be recomputed for example.

BTRFS does support both compression and fsverity, but I don't know if they're compatible...I suspect not.

What's your use case for compressing?

[Winterhuman]

I was mainly curious where the "10MB" came from. From what you're saying though, I guess they meant they literally created something like erofs.img.zst, and that's where the size figure comes from.

I never realised EROFS compression doesn't apply to metadata, but skimming the kernel docs again, it turns out this is actually a design choice: https://erofs.docs.kernel.org/en/latest/features.html#id17

[allisonkarlitskaya]

I literally took the finished image file and piped it through zstd, xz, and gzip with various flags in order to produce that number. They all came in at around 10MB +/- 2.

The final file was not something that we could mount directly, but at the time, we were discussing copying it to a memfd before mounting it, which would have given a good opportunity to decompress it.

[hsiangkao]

https://lore.kernel.org/all/[email protected]/

Looks nice. Might be interesting indeed.

could you share some reference so I could try with mkfs.erofs too?

I can share an erofs file with you if that works for you?

Yes, it works too

[travier]

Zip compressed to pass GitHub checks: ostree.cfs.zip

[hsiangkao]

Zip compressed to pass GitHub checks: ostree.cfs.zip

I've checked the current results:
41000960 ostree.cfs (the original one generated by mkcomposefs)
40505344 ostree.cfs.erofs (by mkfs.erofs -Ededupe -T0 -x1)
36323328 ostree.cfs.erofs.xattrprefix (by mkfs.erofs -Ededupe -T0 -x1 --xattr-prefix="trusted.overlay.metacopy" --xattr-prefix="trusted.overlay.redirect")
and just for reference:
43638784 ostree.cfs.sqfs.uncompressed (by mksquashfs -noI -noX -noappend to disable squashfs metadata/xattr compression)
15847424 ostree.cfs.sqfs.compressed (by mksquashfs -noappend to enable squashfs metadata/xattr compression)

So clearly metadata compression could help on this, but metadata compression will have bad impacts to runtime performance so EROFS didn't consider it as a design goal.
Is it really necessary to implement EROFS metadata compression for composefs use cases? Or just using gzip,zstd-likewise to decompress is enough since the storage is cheap and compared to data the metadata is overally small.

[travier]

Is it really necessary to implement EROFS metadata compression for composefs use cases?

From my perspective, we should prioritize runtime performance. If we can optimize the storage like you're doing with smarter xattr storage then that's good but I don't think we should compromise on runtime performance for lower disk usage.

[hsiangkao]

Is it really necessary to implement EROFS metadata compression for composefs use cases?

From my perspective, we should prioritize runtime performance. If we can optimize the storage like you're doing with smarter xattr storage then that's good but I don't think we should compromise on runtime performance for lower disk usage.

Yeah, metadata compression is bad for random file lookup access for example (due to long tail latencies), and it takes more memory so I don't consider it for now anyway (also it's useless to the original Android use cases.)