Support for advanced filesystem features (setuid/setgid/xattrr/hardlinks/sockets, etc)

[mbiebl]

Version: 0.6.1

Followup to #543

# ls -al /usr/bin/sudo
-rwsr-xr-x 1 root root 147560 Feb  2  2020 /usr/bin/sudo

After the kopia restore

# ls -al /var/tmp/restore/sudo
-rwxr-xr-x 1 root root 147560 Feb  2  2020 /var/tmp/restore/sudo

I assume aside from the SUID bit, SGID and sticky bit are also not (yet) properly supported.

[jkowalski]

Yes, there's a whole bunch of things currently not captured at the filesystem level, including setuid/gid, hardlinks, mount points, sockets, xattr, ACLs, etc.

Currently only files (simple, without alternate streams) and symlinks are supported, and only simple filemode + owner/group are supported.

I'm going to repurpose this issue to track adding the missing pieces and close the others as duplicates.

[mbiebl]

Sounds fine to me, thanks. Wasn't sure if you preferred individual issues or not.

[mbiebl]

Currently only files (simple, without alternate streams) and symlinks are supported

Hm, see #543. When running kopia restore, no symlinks were restored. Browsing the repository via kopia-ui though, I see those files.

[jolly-jim]

Ideally this will include - under Windows - support for:

• File attributes
• ACLs
• Symlinks (file & folder), Junction points & Hard links.

IMO, for the latter, the required behaviour should be to just capture the symlink/junction point/hard link itself, and not it's target.
If there really is a need to capture the target (or more accurately the target's content), then that could be handled with an optional - non default - argument.

[WMP]

If you write this feature kopia will be good tool to backup whole VM, now i can do this with xfs_dump tool.

[dima-altinity]

@jkowalski
very interesting project and I also think that this feature is really missing.
whats the current status of symlinks and hardlinks support?

[jkowalski]

Symlinks and hardlinks are correctly backed up. Both can be restored but hardlinks are restored as individual files and not as hardlinks.

[dima-altinity]

thnx @jkowalski , just would like to clarify how does it work( restore) if there are mutliply hard links to the same file?
lets say I have file /home/user/1.tar.gz ( size 1Gb) and /home/user/link1.gz, /home/user/link2.gz that are hard links to /home/user/1.tar.gz
Scenarios

1. I create and upload backup to S3 for /home/user/ ( how much data will be sent and stored on S3 - 1Gb or 3Gb ?)
2. I create /home/user/link3.gz hardlink that points to the same file /home/user/1.tar.gz. Then upload backup to S3. How much data will sent to S3 during this operation?
3. I restore backup from S3 to local ( backup contains 1 file and 3 hard links). How much data will be downloaded from S3? 1Gb or 4Gb?
4. do I understand right that after restore this backup I'll get 4 files in /home/user/ with total size 4Gb ?

[jkowalski]

1. 1gb because of deduplication
2. 0 (plus minimal overhead for indices and manifests)
3. Physically 1 gb will be downloaded (because of caching) but logically each copy will be independently fetched (so 3gb).
4. That is correct.

[dima-altinity]

how much efforts is required to add full support of hard links ( so that #3 and #4 worked properly)?
is it just changes in code? or changes in architecture also required ?

[jkowalski]

It's mostly some additional restore work, architecture is fine.

[martinpk]

Sockets: until full support is implemented, would you reword this message:

! Ignored error when processing "<filename>": unknown or unsupported entry type

to indicate that (1) the file in question is a socket, and (2) it has not been backed up?

Also in the interim, would you provide a non-default option to ignore sockets, in order to reduce noise?

[dima-altinity]

@jkowalski
does this latest push add full support of hard links? ( backup & restore)

[jkowalski]

No it does not, will reopen.

[yacoob]

One extra comment: it'd be worthwhile to add this limitation to the documentation, to make sure people are aware of this limitation, instead of discovering it at the test - or worse, at restore time.

[seaniedan]

I'm currently snapshotting a disk full of backups made with the excellent RaspiBackup. It has a folder structure of incremental backups using hardlinks. It is taking a very long time to do the first snapshot!

Would it be quicker if I snapshot the latest backup first? Then add the whole disk?

[seaniedan]

In case it's useful for discussion or when writing documentation, this forum page on Duplicati has some good explanations of the difficulties around hardlinks.

[seaniedan]

Hi Jarek, you say supporting hardlinks is "mostly some additional restore work" - Kopia backups with no hardlinks take ~11 seconds for 2.5 TB of files, but a much smaller directory of hardlinks takes ~9 hours. Nothing seems to bring this time down. Is this time expected to come down with an update?

[aniqueta]

Still needed for macOS users. Thanks!

[xeor]

I don't think the fix from @PhracturedBlue is merged into kopia yet? Still a wip maybe?

[PhracturedBlue]

Something screwy going on in github. I don't think I ever touched this code, it somehow got attributed to be due to an unrelated PR.

The fix in question seems to be this one: #2597 which was merged, though I don't think it entirely fixes the issue (based on description).

[miekg]

Hi all, I've found myself in need for support for xattr (on Linux) and was thinking along the following lines, put here to see if this is something I should undertake.

Restic is also in Go and matches license wise and does support xattr. Can I, should I ?, lift and modify that code and prepare a PR here?

[miekg]

this has become: #3643