Symlink errors from tar extraction on Windows

[edmorley]

Using this plugin on Windows resulted in this error:

$ heroku slugs:download -a treeherder-stage
Downloading... ████████████████████████▏  100% 00:00 72.35MB
tar: ./app/.heroku/vendor/lib/libmemcachedutil.so: Cannot create symlink to ‘libmemcachedutil.so.2.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/libhashkit.so: Cannot create symlink to ‘libhashkit.so.2.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libanonymous.so.3: Cannot create symlink to ‘libanonymous.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libanonymous.so: Cannot create symlink to ‘libanonymous.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libplain.so.3: Cannot create symlink to ‘libplain.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libscram.so: Cannot create symlink to ‘libscram.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libgssapiv2.so: Cannot create symlink to ‘libgssapiv2.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libgs2.so.3: Cannot create symlink to ‘libgs2.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libdigestmd5.so: Cannot create symlink to ‘libdigestmd5.so.3.0.0’: No such file or directory
tar: ./app/.heroku/vendor/lib/sasl2/libcrammd5.so: Cannot create symlink to ‘libcrammd5.so.3.0.0’: No such file or directory
tar: ./app/.heroku/python/share/man/man1/python.1: Cannot create symlink to ‘python2.1’: No such file or directory
tar: ./app/.heroku/python/bin/python-config: Cannot create symlink to ‘python2-config’: No such file or directory
tar: ./app/.heroku/python/bin/python: Cannot create symlink to ‘python2’: No such file or directory tar: ./app/.heroku/python/bin/python2-config: Cannot create symlink to ‘python2.7-config’: No such file or directory
tar: ./app/.heroku/python/bin/python2: Cannot create symlink to ‘python2.7’: No such file or directory
tar: ./app/.heroku/python/lib/libsqlite3.so: Cannot create symlink to ‘libsqlite3.so.0.8.6’: No such file or directory
tar: ./app/.heroku/python/lib/pkgconfig/python.pc: Cannot create symlink to ‘python2.pc’: No such file or directory
tar: Exiting with failure status due to previous errors
 !    Command failed: tar -xf treeherder-stage/slug.tar -C treeherder-stage
 !    tar: ./app/.heroku/vendor/lib/x86_64-linux-gnu/libmysqlclient.so: Cannot create symlink to
 !    ‘libmysqlclient.so.20’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/libmemcachedutil.so: Cannot create symlink to
 !    ‘libmemcachedutil.so.2.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/libhashkit.so: Cannot create symlink to ‘libhashkit.so.2.0.0’:
 !    No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libanonymous.so.3: Cannot create symlink to
 !    ‘libanonymous.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libanonymous.so: Cannot create symlink to
 !    ‘libanonymous.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libplain.so.3: Cannot create symlink to
 !    ‘libplain.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libscram.so: Cannot create symlink to ‘libscram.so.3.0.0’: !    No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libgssapiv2.so: Cannot create symlink to
 !    ‘libgssapiv2.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libgs2.so.3: Cannot create symlink to ‘libgs2.so.3.0.0’:
 !    No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libdigestmd5.so: Cannot create symlink to
 !    ‘libdigestmd5.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/vendor/lib/sasl2/libcrammd5.so: Cannot create symlink to
 !    ‘libcrammd5.so.3.0.0’: No such file or directory
 !    tar: ./app/.heroku/python/share/man/man1/python.1: Cannot create symlink to ‘python2.1’: No
 !    such file or directory
 !    tar: ./app/.heroku/python/bin/python-config: Cannot create symlink to ‘python2-config’: No
 !    such file or directory
 !    tar: ./app/.heroku/python/bin/python: Cannot create symlink to ‘python2’: No such file or
 !    directory
 !    tar: ./app/.heroku/python/bin/python2-config: Cannot create symlink to ‘python2.7-config’: No  !    such file or directory
 !    tar: ./app/.heroku/python/bin/python2: Cannot create symlink to ‘python2.7’: No such file or
 !    directory
 !    tar: ./app/.heroku/python/lib/libsqlite3.so: Cannot create symlink to ‘libsqlite3.so.0.8.6’:
 !    No such file or directory
 !    tar: ./app/.heroku/python/lib/pkgconfig/python.pc: Cannot create symlink to ‘python2.pc’: No
 !    such file or directory
 !    tar: Exiting with failure status due to previous errors
 !

This is because Windows doesn't support symlinks.

Possible solutions:

1. Default to not extracting the archive (see Allow downloading a slug without extracting it #2) and let the user deal with how they want to extract it.
2. Pass the --dereference option to tar on Windows (only), to ensure symlinks are followed and converted to real files.

This was using:
heroku-cli/6.11.17-91bdf0b (windows-x64) node-v7.10.0
heroku-slugs 1.0.3

[edmorley]

Ah so --dereference is just for when the tar is created, not for extraction. Also, the error message from the first invocation isn't quite as bad as it looks -- the extraction didn't abort halfway through, there were just a small number of skipped files.

It appears that GNU tar (for at least the MSYS2 package of it) does partially handle filesystems that don't support symlinks, in that whilst extracting it does replace the symlink with a copy of the full file, if the file already exists on the filesystem.

The problem with this is that it's dependant on the order of the files in the tar, since if the symlink is extracted first, the target won't yet exist (this isn't a problem on linux since dangling symlinks are allowed). This comes from the fact that tar is intended as a streaming format, though perhaps GNU tar should still try and do the right thing when in seekable mode (but fixing GNU tar is beyond the scope of this ticket).

As such, a workaround is to run the tar extraction multiple times, which will eventually succeed without errors once all the symlink targets have been created. (Running just twice is not sufficient, since there may be symlink chains more than 1 level deep - eg: python -> python2 -> python2.7)

I tried looking into some of the nodejs tar packages, such as tar and tar-fs to see if they fared any better - however I think at best they'll have the same issue, and at worst they may not even support the "convert symlink to full file if it already exists" behaviour of GNU tar (given isaacs/node-tar#38).

Judging from various stack overflow and github issues I've read, one of the problems here is that tools/libraries don't treat this as an issue they should address since they see archives that contain symlinks as not portable, and as such the "fault" of the creator of the archive. (For Heroku's case, containing symlinks is in my opinion legitimate; downloading the slugs for inspection on Windows is just an unfortunate edge case.)

As such, the options here seem to be:

1. Just add a note to the README saying "on Windows there may be symlink errors, either ignore or re-extract using tar multiple times to work around".
2. Try to detect this scenario and have the command output such a message if errors occurred (and possibly suppress/tidy up the error output)
3. Same as (2) but have the tool automatically repeat extraction up to N times (where N is perhaps <=3)

[duanehutchins]

This issue still happens. Is there any fix?

[edmorley]

One workaround is to enable symlink support on Windows:
https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

And then if using MSYS2, ensure MSYS=winsymlinks:native is set in the environment.
After that, the heroku slugs:download command just works.

Another is to just accept the inevitable and use WSL...
(switching from MSYS2 to WSL is first on my list when I get my new laptop 😆)

[duanehutchins]

ah, this is gold. Thank you!

[davidhinton23]

I arrived here with the following issue:

tar: <LIB_NAME>.so: Cannot create symlink to ‘<LIB_NAME>.so.9.2.0’: No such file or directory
tar: Exiting with failure status due to previous errors

The workaround suggested by @edmorley:

As such, a workaround is to run the tar extraction multiple times, which will eventually succeed without errors once all the symlink targets have been created. (Running just twice is not sufficient, since there may be symlink chains more than 1 level deep - eg: python -> python2 -> python2.7)

is sufficient to alleviate my pain - i.e. A second identical call to tar immediately following the first yielded a complete, error-free extraction.

Thanks @edmorley!