Bug: pylsp + ruff: config files ignored in subdirectories

[jhossbach]

Description

TL;DR: ruff needs ruff.toml / pyproject.toml files with its configuration to be placed inside the JP_LSP_VIRTUAL_DIR in order to find them.

This is coming from python-lsp/python-lsp-ruff#102, the relevant comment is python-lsp/python-lsp-ruff#102 (comment).
The issue is that python-lsp-ruff, the ruff extension for python-lsp-server passes in the filename in order for ruff to apply the correct configurations (whether it should ignore certain errors for this file etc.). This filename is used to also figure out ruff configuration files it should apply in addition to the options passed directly.
But ruff cannot find the relevant config files if the jupyter notebook is in a subdirectory of the CWD:

.
├── .virtual_documents
│   └── foo
│       └── bar.ipynb
└── foo
    ├── bar.ipynb
    └── ruff.toml

For ruff to find the ruff.toml it would need to traverse up to . and then go down again into ./foo/ which it does not do.

The easy solution IMO would be to copy/link ruff.toml and pyproject.toml files into the virtual_documents directory, or to give ruff the actual file path rather than the path to the notebook inside the virtual_documents directory.

Reproduce

Shamelessly borrowed from the original issuer python-lsp/python-lsp-ruff#102 (comment)

1. uv venv && . .venv/bin/activate
2. uv pip install jupyterlab jupyterlab-lsp python-lsp-ruff ruff
3. Create a pylsp config like the following:

$ cat << EOF >.venv/share/jupyter/lab/settings/overrides.json
{
  "@jupyter-lsp/jupyterlab-lsp:plugin": {
    "language_servers": {
      "pylsp": {
        "priority": 50,
        "serverSettings": {
          "pylsp.plugins.flake8.enabled": false,
          "pylsp.plugins.pycodestyle.enabled": false,
          "pylsp.plugins.pyflakes.enabled": false,
          "pylsp.plugins.pylint.enabled": false,
          "pylsp.plugins.ruff.enabled": true,
          "pylsp.plugins.ruff.formatEnabled": true,
          "pylsp.plugins.ruff.lineLength": 120,
          "pylsp.plugins.yapf.enabled": false
        }
      }
    }
  }
}
EOF

3. jupyter lab --notebook-dir=/tmp
4. Create a top-level notebook with the code import os and confirm that it triggers error F401, as
5. Create a top-level ruff.toml with ignore = ["F401"] in the [lint]:

[lint]
ignore = ["F401"]

7. Reload the notebook from disk and confirm that error F401 is no longer triggered, as
8. Close the notebook
9. Move both the notebook and the ruff.toml into any subdirectory
10. Open the notebook from inside the subdir
11. Observe that the ruff.toml is ignored and error F401 is triggered
12. The same bug occurs when using pyproject.toml instead of (or in addition to) ruff.toml

Expected behavior

Context

This context reflects my setup, but the issue was observed on other machines with (likely) different setups.

• Operating System and version: Linux 6.15.9-arch1-1
• Browser and version: Mozilla Firefox 141.0.3
• JupyterLab version: 4.4.5
• Language server and version: python-lsp-server 1.12.2, python-lsp-ruff 2.2.2, ruff 0.12.8

Required: Jupyter server paths

$ jupyter server extension list
Config dir: $HOME/.config/jupyter

Config dir: /usr/etc/jupyter

Config dir: /usr/local/etc/jupyter

Required: installed lab extensions

 $ jupyter labextension list
`sys_prefix` level settings are read-only, using `user` level for migration to `lockedExtensions`
JupyterLab v4.4.5
/usr/share/jupyter/labextensions
        jupyterlab_code_formatter v3.0.2 enabled OK (python, jupyterlab_code_formatter)
        catppuccin_jupyterlab v0.2.4 enabled OK (python, catppuccin_jupyterlab)
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
        @jupyter-lsp/jupyterlab-lsp v5.2.0 enabled OK (python, jupyterlab-lsp)
Disabled extensions:

@jupyterlab/completer-extension:base-service

@jupyterlab/fileeditor-extension:language-server

@jupyterlab/lsp-extension:settings

@jupyterlab/notebook-extension:language-server