Flycheck with Tramp over SSH not working for Python files

[MemoWalk]

Checklist

• [ x] I have checked existing issues for potential duplicates before creating this one.
• [ x] I have read the Troubleshooting guide.

Bug description

Flycheck seems not to work correctly with Tramp Mode on remote files. My setup etc. is described in this post on Emacs Stackexchange.

Steps to reproduce

• [x ] I have read https://emacs.stackexchange.com/questions/28429/how-do-i-troubleshoot-emacs-problems

Confer link.

Expected behavior

Someone from the ´[email protected]´ offered help (you may contact him, as I understood) and pointed out:

• "[...] it turns out [flycheck.el] doesn't support remote processes. It calls call-process' (shall be process-file'), and start-process' (shall be start-file-process')."
• "[...] pylint.el doesn't call any external process. [...] For example, it
  uses (aref (tramp-dissect-file-name filename) 3) [...]; [...] it shall use (file-local-name filename) .

System configuration

Confer link.

Emacs configuration:

• [ x] Plain Emacs / Custom configuration
• Spacemacs
• Doom Emacs
• Other shared configuration

Additional notes

Add any other context about the problem here.

[cpitclaudel]

Hi there, and thanks for the report!

We don't officially support checking over tramp, AFAIK (see #54), though I'd love for us to do this reliably. Is this something you would be interested in working on?

Also, this part: pylint.el doesn't call any external process; can you point to that discussion? I didn't see it in your link.

[albinus]

I might be able to help with Tramp integration. Since I'm not a flycheck user, I'd need some guidance. At least a short recipe for testing.

[cpitclaudel]

Thanks a lot @albinus , your help would be much appreciated :) We need someone to drive the process, and I'm swamped with work these days, but once we have a volunteer it should be quick and easy.

In my experience, porting packages to work with Tramp is very easy; I've done it a few times before, including for modes that use Flycheck's more general APIs (Flycheck has two APIs: one for command-line checkers that respond on stdin, and one for subprocesses, which is useful to chache results from run to run, with LSP, etc.). So really the only thing to do here is to change the simplified Flycheck API for command-line checkers to use the right process-starting calls, and to adjust the other places that start processes (to check if a particular checker is available, for example). I recently did some refactoring of that code to make all thee calls go through one function, so it shouldn't be hard.

[MemoWalk]

Hi There,
I am an Emacs Lisp Newbie and can spare only little time, but little time it is. I would be happy to do the "boring" stuff. Albeit I have programmed in clojure and common lisp, I am not familiar with elisp (yet). So if someone tells me what to change into what (your answer, @cpitclaudel already points in a direction), I think I can do the testing and the coding.

[borkdude]

Slightly related:

I'm also interested in supporting flycheck-clj-kondo, a linter for Clojure, in tramp-mode:

Preferably the clj-kondo linter runs on the remote, not locally. Although it kind of works locally, it misses out on configuration files on the remote machine which makes linting less accurate.

[cpitclaudel]

I'm also interested in supporting flycheck-clj-kondo, a linter for Clojure, in tramp-mode:

That would be great.

I would be happy to do the "boring" stuff.

Nice! ELisp programming is never boring :) The right way to do this would be to get familiar with tramp by reading its manual first; that will tell you which APIs need to change, I think. Then you'd audit the Flycheck code to determine which places need changes, and change these. And finally you'd write tests to make sure that things work.

A few difficulties that will likely pop up: looking in the right places for configuration files; locating executables (and modules for python, R, and probably a few other checkers); invoking processes the right way. We might also want to set appropriate timeouts here and there, since Flycheck currently assumes that disk access and sending stuff to processes is fast, so it does it in a blocking way. If it can hang, we might want to be more careful.

[carrete]

I believe this applies to #1777 as well. In flycheck-checker-shell-command this:

(concat command " < " (shell-quote-argument (buffer-file-name)))

should look something like:

(concat command " < " (shell-quote-argument (file-remote-p (buffer-file-name) 'localname)))

for remote files. I understand this isn't a complete nor proper solution. I simply mean this as an example.

With this change, running C-c ! C-c on a TRAMP buffer produces:

-*- mode: compilation; default-directory: "/docker:localhost:/opt/kibit-runner/src/kibit_runner/" -*-
Compilation started at Tue Sep 22 19:04:25

clj-kondo --lint - --lang clj --cache < /opt/kibit-runner/src/kibit_runner/cmdline.clj
<stdin>:4:6: warning: namespace clojure.test is required but never used
linting took 29ms, errors: 0, warnings: 1

Compilation exited abnormally with code 2 at Tue Sep 22 19:04:25

Without:

-*- mode: compilation; default-directory: "/docker:localhost:/opt/kibit-runner/src/kibit_runner/" -*-
Compilation started at Tue Sep 22 19:06:47

clj-kondo --lint - --lang clj --cache < /docker\:localhost\:/opt/kibit-runner/src/kibit_runner/cmdline.clj
/bin/sh: line 2: /docker:localhost:/opt/kibit-runner/src/kibit_runner/cmdline.clj: No such file or directory

Compilation exited abnormally with code 1 at Tue Sep 22 19:06:48

Unfortunately, running C-c ! c still produces:

Suspicious state from syntax checker clj-kondo-clj: Flycheck checker clj-kondo-clj returned non-zero exit code 127, but its output contained no errors: 
Try installing a more recent version of clj-kondo-clj, and please open a bug report if the issue persists in the latest release.  Thanks!
condition-case: Process flycheck-clj-kondo-clj not running

I'm hoping someone more knowledgeable with flycheck.el could point me to the other places I should look to come up with a complete and proper solution. Thanks!

[albinus]

Tom Vaughan <[email protected]> writes: Hi Tom,

I believe this applies to #1777 as well. In flycheck-checker-shell-command this: (concat command " < " (shell-quote-argument (buffer-file-name))) should look something like: (concat command " < " (shell-quote-argument (file-remote-p (buffer-file-name) 'localname))) for remote files. I understand this isn't a complete nor proper solution. I simply mean this as an example.

Yes, this is the recommended way to go. However, if (buffer-file-name) returns a local file name, it doesn't work, because (file-remote-p ...) would return nil. Therefore, we have (since Emacs 26.1) the function file-local-name. So your change should be (concat command " < " (shell-quote-argument (file-local-name (buffer-file-name)))) Here and everywhere, a file name is use as argument of a (possibly remote) process. Best regards, Michael.

[carrete]

Awesome. Thank you @albinus. This is exactly the sort of feedback I was fishing for

[sugitach]

I got similar case on flycheck for python.
flycheck reported ".pylint" was not found. It look like this.

Suspicious state from syntax checker python-pylint: Flycheck checker python-pylint returned 1, but its output contained no errors: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/runpy.py", line 192, in run_module
    fname, loader, pkg_name)
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/site-packages/pylint/__main__.py", line 7, in <module>
    pylint.run_pylint()
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/site-packages/pylint/__init__.py", line 16, in run_pylint
    Run(sys.argv[1:])
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/site-packages/pylint/lint.py", line 1307, in __init__
    linter.read_config_file()
  File "/Users/sugita/.pyenv/versions/2.7.17/lib/python2.7/site-packages/pylint/config.py", line 646, in read_config_file
    raise IOError("The config file {:s} doesn't exist!".format(config_file))
IOError: The config file /scpx:sugita-linux.local:/home/sugita/work/auto/cdn-test/util/testtool/fabrictest/.pylintrc doesn't exist!

Is this the same cause?

[timlod]

I get a similar error with flycheck-pyflakes when executing `C-c ! C-c:

-*- mode: compilation; default-directory: "/ssh:myserver:/home/tim/" -*-
Compilation started at Sun Oct 11 09:54:12

/home/tim/miniconda3/bin/pyflakes /ssh\:myserver\:/home/tim/test.py
/bin/sh: line 2: /home/tim/miniconda3/bin/pyflakes: No such file or directory

Compilation exited abnormally with code 127 at Sun Oct 11 09:54:13

myserver is the remove server I'm using (where the remote file is located).
I use python virtualenvs on that server, and installed pyflakes in both base and virtualenv pythons just in case.

.../miniconda3/bin/ is where my local python, and thus pyflakes, lives.

In "normal" flycheck usage, ie. when I just navigate and change a remote buffer, I get the following error message in the minibuffer:

Suspicious state from syntax checker python-pyflakes: Flycheck checker python-pyflakes returned 1, but its output contained no errors: /ssh:tau:/home/tim/flycheck_test.py: No such file or directory

Try installing a more recent version of python-pyflakes, and please open a bug report if the issue persists in the latest release.  Thanks!

I also tried changing the pyflakes command to the one installed on my remote server - flycheck then runs, but throws a fatal error that the module (which I'm editing) is not found.

[carrete]

Please see #1842

[FraPsiZeta]

Any news about this? It's unfortunate this issue got so little attention, it completely prevents the use of TRAMP with python.

@carrete I've tried your fix but unfortunately all checkers gave one of the following error:

• executable: Not found
• `pylint' module: Missing; sys.path is nil

[xelibrion]

Is anyone able to verify and merge #1842?

[carrete]

Is anyone able to verify and merge #1842?

@xelibrion I've been using this daily since I posted the pull-request and everything seems to work ok. The tests pass for me when I run them locally, but they fail on Travis CI. I've rebased my branch several times in the hopes that this might magically fix things, alas no such luck. I've sort of given up at this point

[spencerahill]

Another +1 on getting #1842 out from a user that would be very grateful :)

[kantholtz]

Hi! Thanks for working on getting tramp support. Any reason why #1922 has not been merged yet?

[cpitclaudel]

Hi! Thanks for working on getting tramp support. Any reason why #1922 has not been merged yet?

Yes: no-one has (had time to / volunteered to) do a thorough review of it

[cpitclaudel]

(To be clear, I'm very thankful to both @carrete and @yfyyfy, I'm just not finding the time to do a proper review; if someone else can read through the proposed implementation and comment, that would be wonderful. Maybe @carrete can look at the extensions that @yfyyfy made, to start with?)

[kantholtz]

Thanks for the quick response! No pressure; I was just curious because I have had problems with python-flake8 (and friends) over tramp.

[spetryk]

Thanks for setting this up! I've been trying to use the latest commit on #1842 . However, running into some issues when trying to run pylint. TL;DR is that pylint creates temporary flycheck files, yet it takes the path of the remote file I'm editing (without the /sshx:... prefix) and tries to look for it on my local filesystem.

What's the expected location for the executables, local or remote? I had to manually set flycheck-python-pylint-executable to my local /path/to/local/bin/pylint; otherwise, I would get a Missing; sys.path is nil error when running flycheck-verify-setup. Now, things run fine locally, and this is the flycheck-verify-setup I get for a remote python file:

First checker to run:

  python-pylint (explicitly selected)
    - may enable:         yes
    - executable:         Found at /path/to/local/bin/pylint
    - configuration file: Found at "/path/to/local/.pylintrc"
    - next checkers:      python-mypy
...
Flycheck Mode is disabled. Use C-u M-x flycheck-disable-checker to
enable disabled checkers.

I thought it was a little odd that Flycheck Mode was disabled, despite having set global-flycheck-mode, and flycheck running automatically on a local file. When I try to run flycheck-mode on my remote file, say /sshx:..../my/remote/home/test.py, I get the error:

Error while checking syntax automatically: (file-error "Creating directory" "Permission denied" "/my/remote")
as pylint is trying to create the temporary /my/remote/home/flycheck_test.py file, yet it's trying to look for the directory on my local filesystem.

Does anyone know what's going on here? I've tried to use a remote executable, which would indeed run pylint/flycheck on a remote file - yet would be extremely laggy when I would edit the file, as it kept repeatedly opening tramp connections. Any help would be much appreciated!