Packaging with "tar.exe" on Windows

[dominiksta]

Hi,

I am trying out Eldev for the first time and I am really liking it so far. However, I think I may have stumbled upon a bug when trying to create a tarball on windows - meaning I cannot build a package.

Here is the output of eldev -vt package:

Started up on Thu Sep 30 16:06:59 2021
Running on GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)
 of 2021-03-26
Project directory: `c:/Users/dominik/.emacs.d/straight/repos/mvtn.el/'
No file `c:/Users/dominik/.eldev/config', not applying user-specific configuration
Loading file `Eldev'...
Using package archive `melpa-stable' at `https://stable.melpa.org/packages/' with priority 200
Using package archive `melpa-unstable' at `https://melpa.org/packages/' with priority 100
Reading target dependencies from file `.eldev/27.2/target-dependencies.build'...
Saving target dependencies to file `.eldev/27.2/target-dependencies.build'...
No file `Eldev-local', not customizing build
Executing command `package'...
Contents of package archive `melpa-stable' has been fetched already
Contents of package archive `melpa-unstable' has been fetched already
Not fetching contents of other archive(s) as redundant
All project dependencies (including those for set `build') have been installed already or are local
Loading file `mvtn-autoloads.el'
Generating build target list for standard fileset `main'
Ignored potential target: `mvtn-pkg.elc' (1)
Reading target dependencies from file `.eldev/27.2/target-dependencies.build'...
Building plan: `dist/mvtn-0.1.tar', `:package'
PACK     -> dist/mvtn-0.1.tar
Packaging the following files: `mvtn-ag.el', `mvtn-compat.el', `mvtn-file-helpers.el', `mvtn-link-buttons.el', `mvtn-pkg.el', `mvtn-rg.el', `mvtn-tag-addons.el', `mvtn-templates.el', `mvtn.el' (9)
Full command line to create package tarball:
  c:/Program Files/Git/usr/bin/tar.exe -cf c:/Users/dominik/.emacs.d/straight/repos/mvtn.el/dist/mvtn-0.1.tar mvtn-0.1/mvtn-ag.el mvtn-0.1/mvtn-compat.el mvtn-0.1/mvtn-file-helpers.el mvtn-0.1/mvtn-link-buttons.el mvtn-0.1/mvtn-pkg.el mvtn-0.1/mvtn-rg.el mvtn-0.1/mvtn-tag-addons.el mvtn-0.1/mvtn-templates.el mvtn-0.1/mvtn.el

`tar' output (ran from directory `c:/Users/dominik/AppData/Local/Temp/eldev-packaging-088kme'):
Saving target dependencies to file `.eldev/27.2/target-dependencies.build'...

Failed to create package tarball `dist/mvtn-0.1.tar'
Finished erroneously on Thu Sep 30 16:07:02 2021

What seems to be happening is that "tar.exe" does not like dealing with paths prefixed with c:/ or generally any drive letter. At least this simple test command from powershell implies this:

PS Desktop> &"c:/Program Files/Git/usr/bin/tar.exe" -cf "test.tar" "test.txt"
# works
PS Desktop> &"c:/Program Files/Git/usr/bin/tar.exe" -cf "c:/users/dominik/desktop/test.tar" "test.txt"
/usr/bin/tar: Cannot connect to c: resolve failed

I have tested this with the "tar.exe" from both cygwin and git bash - they both show the same error. Theoretically, it should absolutely be possible to build a package from wsl, but this is an inconvenience for someone like me who primarily uses native Windows tooling (please don't bully me ;) ).

[doublep]

Eldev now supports Windows, but I personally don't use it and don't even have access to it. Maybe @juergenhoetzel or @ikappaki, who actually added Windows support, can help. Maybe we could just try using non-absolute filename for the target (.tar) file.

By the way, it is better not to use -vt. These options are not additional and the latter (-t) simply overrides the former (-v). In this case result is fine, but e.g. -tv would be the same as only -v, making -t have no effect.

[ikappaki]

Hi @dominiksta,

It appears your are picking the tar executable that comes bundled with your git installation:

Full command line to create package tarball:
  c:/Program Files/Git/usr/bin/tar.exe -cf c:/Users/dominik/.emacs.d/straight/repos/mvtn.el/dist/mvtn-0.1.tar mvtn-0.1/mvtn-ag.el mvtn-0.1/mvtn-compat.el mvtn-0.1/mvtn-file-helpers.el mvtn-0.1/mvtn-link-buttons.el mvtn-0.1/mvtn-pkg.el mvtn-0.1/mvtn-rg.el mvtn-0.1/mvtn-tag-addons.el mvtn-0.1/mvtn-templates.el mvtn-0.1/mvtn.el

which does not understand drive letters.

Can you jingle your PATH env variable to pick up the tar version instead that comes with Windows 10 at C:\Windows\System32\tar.exe?

If you call where tar from the command prompt it will give you a list of alll the tar executables available on your system.

Thanks

[juergenhoetzel]

which does not understand drive letters.

The root cause of the problem is the fact that GNU tar interprets tar archives with colons as network addresses.

Can you jingle your PATH env variable to pick up the tar version instead that comes with Windows 10 at C:\Windows\System32\tar.exe?

I you don't want to customize your system PATH, you can also setup a Eldev-localfile like this:

(setf eldev-tar-executable "C:/Windows/system32/tar.exe")

[dominiksta]

Oh wow I was not aware that Windows ships with a tar executable. I thought I had to use some third party tool like Git Bash or Cygwin to have a tar executable. The suggested included executable works! Thank you.

For other potential users like me, would it maybe make sense to include a word or two about this in the documentation? Alternatively, would it not make some sense to change the definition of eldev-tar-executable?

(defun eldev-tar-executable (&optional not-required)
  "Find `tar' executable.
See also variable `eldev-tar-executable'."
  (eldev-find-executable
   eldev-tar-executable not-required
   (if (eq system-type 'windows-nt) "C:/Windows/System32/tar.exe"
     (or (executable-find "gtar") (executable-find "tar")))
    "Cannot find `tar' program"))

[doublep]

you can also setup a Eldev-local file

Even better would be to write it in ~/.eldev/config, so that it is system-global for you (unless you can modify $PATH, of course).

The root cause of the problem is the fact that GNU tar interprets tar archives with colons as network addresses.

I'd say it is a bug in GNU tar then and should be reported.

For other potential users like me, would it maybe make sense to include a word or two about this in the documentation?

It feels rather pointless, I doubt anyone will search for this in documentation.

Alternatively, would it not make some sense to change the definition of eldev-tar-executable?

Feels a bit backwards, there is system variable $PATH for a reason, I'd say.

We can probably just use file-relative-name in the following line, maybe on Windows only:

(eldev-call-process (eldev-tar-executable) `("-cf" ,(expand-file-name package-target eldev-project-dir) ,@(nreverse files-to-tar))

It will theoretically not always be enough because of eldev-package-output-dir, which could point to another drive, but likely good enough in practice. Alternatively, we could add a heuristic and print a hint if tar gives this error (can we somehow distinguish it from other errors?). But it would be nice if GNU tar got fixed in addition to any workarounds we add.

[juergenhoetzel]

I'd say it is a bug in GNU tar then and should be reported.

I guess its a feature. GNU-Info tar page:

   If the archive file name includes a colon (':'), then it is assumed
to be a file on another machine.  If the archive file is
'USER@HOST:FILE', then FILE is used on the host HOST.  The remote host
is accessed using the 'rsh' program, with a username of USER.  If the
username is omitted (along with the '@' sign), then your user name will

which can be disabled:

      --force-local          archive file is local even if it has a colon

It's pretty useless these days and it leads to many problems under windows nowadays:

package-build--create-tar: don't fiddle with windows filenames

[doublep]

Gah, what a mess. I guess reporting is pointless then, even if the supposed feature is stupid when "USER" is omitted and "HOST" consists of a single letter (i.e. just a normal Windows path).

Anyway, I will accept practically any improvement/workaround for Eldev (e.g. using relative path after -cf whenever possible, documentation additions, special hints if it is possible to say that it's likely this error and so on) except for explicitly fixing on "C:/Windows/System32/tar.exe". I just feel it would be too dirty, as everything else is looked up through $PATH.

[ikappaki]

How about we introduce a new eldev doctor command to run diagnostics checks on current setup/project for identifying potential issues and providing with suggestions how to fix, à la brew doctor?

Check your system for potential problems. Will exit with a non-zero status if any potential problems are found. Please note that these warnings are just used to help the Homebrew maintainers with debugging if you file an issue. If everything you use Homebrew for is working fine: please don’t worry or file an issue; just ignore this.

Happy to start by implementing something for the case at hand.

[doublep]

Sounds interesting, but would be nice to have at least 3-5 problems it could find, and preferably not only Windows-specific. Also, if you start this, please create a separate file, e.g. eldev-doctor.el, since it is going to potentially become large, but not widely used, so why slow down normal startup. I actually plan to do it for a few commands, for now this is done for eldev-init.

By the way, proposal from my last comment still holds. I just don't want to do it myself, as I don't even have access to Windows.

[ikappaki]

Sounds interesting, but would be nice to have at least 3-5 problems it could find, and preferably not only Windows-specific. Also, if you start this, please create a separate file, e.g. eldev-doctor.el, since it is going to potentially become large, but not widely used, so why slow down normal startup. I actually plan to do it for a few commands, for now this is done for eldev-init.

Great, I will consider it. Although I'm might not be able to find 3-5 problems to include as a starting point, I could perhaps include some diagnostics information (eldev/emacs version build/project artifacats) that could be useful for users to include in their issue reports. Would this perhaps be enough for an initial release?

By the way, proposal from my last comment still holds. I just don't want to do it myself, as I don't even have access to Windows.

I am more of the opinion that we should default tar to "C:/Windows/System32/tar.exe" on MS-Windows really, since anything else is asking for trouble (there is no decent native tar package I can find out there for MS-Windows). And this is why I haven't followed up on this thread until eldev doctor crossed my mind, which I think it will become a a valuable support tool far beyond this case.

Thanks!

[doublep]

Great, I will consider it. Although I'm might not be able to find 3-5 problems to include as a starting point, I could perhaps include some diagnostics information (eldev/emacs version build/project artifacats) that could be useful for users to include in their issue reports. Would this perhaps be enough for an initial release?

Would be an improvement, but still I'd like to see at least one generally (i.e. not just Windows) useful item from the start. As an idea, have a look through closed issues, maybe also those that are not bugs, but e.g. misconfigurations. Maybe you could detect some of those.

I am more of the opinion that we should default tar to "C:/Windows/System32/tar.exe" on MS-Windows really, since anything else is asking for trouble (there is no decent native tar package I can find out there for MS-Windows).

Sorry, no. I feel like if someone put something on your $PATH (even if unknowingly), they should see it used by other tools. We can add a workaround for a known problem (don't use absolute paths for tar on Windows if possible), but I don't want to ignore $PATH changes. If anything, lobby Git and Cygwin into not using GNU Tar, because it is stupid and there is a better alternative.

[doublep]

@ikappaki: I have started command doctor in commit e6a88f4. If you feel like implementing some Windows-specific tests, that would be great.

[ikappaki]

@ikappaki: I have started command doctor in commit e6a88f4. If you feel like implementing some Windows-specific tests, that would be great.

Hi @doublep, this is great, I will have a look to diagnose this issue on MS-Windows, thanks