Can't use eglot with latest Emacs & straight, due to project.el location

[garyo]

I install eglot via straight, and it's been working fine for a long time, until I just updated to the latest emacs master branch and pulled all the latest straight packages. Now emacs starts OK, but when I invoke eglot I get an error: "funcall: Feature provided by other file: project". This comes from require-with-check which looks to see if the feature project was loaded, using load-history. In my case, load-history does not contain `project.

In fact just calling (require-with-check 'project) fails with the same error. I think it's using the built-in version rather than what's loaded from straight.
I checked, and the built-in version is byte-for-byte identical with the version in straight/repos, but I think require-with-check checks the file path.
I also deleted straight.el's copy of the project repo and restarted, and straight clones it again, but I still get the error.

If I debug into require-with-check, I see it calls locate-file on project in load-path and that returns ~/.config/emacs/straight/build/project/project.elc but that is not found in load-history. That only contains "/Applications/Emacs.app/Contents/Resources/lisp/progmodes/project.elc" (or similar on Windows).

I looked at #551 and other related issues, but they don't seem quite the same.

Here's a test case. Load this file into an emacs -Q session:

;;; Set up package system -- straight.el
(defvar bootstrap-version)
(or (boundp 'native-comp-deferred-compilation-deny-list)
    (setq native-comp-deferred-compilation-deny-list '()))

(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

;;; Meta-package system: use-package. Auto-installs and configures packages.
(straight-use-package 'use-package)
(defvar straight-use-package-by-default)
(setq straight-use-package-by-default t) ; make use-package use straight

;;; code to test
(set-variable 'debug-on-error t)
(use-package projectile)
(use-package eglot)
(require-with-check 'project)

and you'll see the error.

If you add (use-package project) before projectile, then it works OK.

[raxod502]

Huh... interesting... let me have a look at what is going on there. Thanks for the ideas on debugging.

[raxod502]

Ok, I see that require-with-check is a very new function, implemented last month in Emacs core: https://gitlab.com/ganserla/emacs/-/commit/2a8e6c8c84ed33674e525625644d5ce84ee8c59a

Maybe there is an edge case when the same package is provided from multiple locations, that is not taken into account properly by one system or the other. I'll build a more recent Emacs snapshot and try it out since I have not used this function before.

[raxod502]

Yes, okay, this makes sense. The issue is that projectile loads project, but doesn't declare it as a dependency in this line:

https://github.com/bbatsov/projectile/blob/0163b335a18af0f077a474d4dc6b36e22b5e3274/projectile.el#L9

So straight.el does not know that it needs to install project, and the built-in version gets loaded. But then eglot does declare project as a dependency:

https://github.com/joaotavora/eglot/blob/db91d58374627a195b731a61bead9b4f84a7e4bc/eglot.el#L10

Which causes straight.el to install project. But the other version was already loaded, so require-with-check correctly identifies that there is a confusing situation.

I think the correct solution here is for projectile to declare a dependency on project. But, straight.el can be improved by reporting a warning if a package is made available when a different version of that same package has already been loaded into the current Emacs session. I'll file another issue to track that improvement, and link it below.

[raxod502]

We can leave this issue open until Projectile is updated to declare the dependency, and new users will not encounter this problem anymore.

The workaround is as you observed; (straight-use-package 'project) or equivalent before running any code that would (require 'project).

[alphapapa]

Forgive the intrusion, but I wonder if this is another instance of the problem reported at these issues, relating to a master version of the map library being installed for a non-master build of Emacs, which fails since it depends on the version of pcase from master, which has associated changes from https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=806759dc0a6a3b049ce35d0497011464e5fc4dcb:

• Any idea how to fix burly-bookmark-handler: Symbol’s value as variable is void: _? alphapapa/burly.el#64
• Newer version of map library is required for pcase expansion alphapapa/bufler.el#70 (comment)

[alphapapa]

Yes, look here: alphapapa/burly.el#64 (comment) The repo at https://github.com/emacs-straight/map.git is importing map.el from Emacs master, which is ending up installing changes to be released in Emacs 30 into configurations running earlier Emacs versions, which is producing "chimera" configurations that are broken. And since the map.el version header still shows 3.3.1, users think they have map v3.3.1 installed, same as the one available from GNU ELPA, while in fact they have map.el from Emacs master ("30.0.50") which will probably be tagged as map v3.4 someday.

[garyo]

It may indeed be related, but not quite the same. In my case both project.el versions are identical; it's the new require-with-check function in emacs itself which checks the on-disk location of the required file that causes trouble.

[progfolio]

@raxod502:

So straight.el does not know that it needs to install project, and the built-in version gets loaded.

We could prevent this and similar issues if straight didn't install external versions of core packages unless they were explicitly requested by the user.
Elpaca offers the following, which ignores built-in packages and can be extended by the user if needed:

(defcustom elpaca-ignored-dependencies (mapcar #'car package--builtin-versions)
  "List of IDs which are not installed unless the user explicitly requests them."
  :type '(repeat symbol))

@alphapapa:

[S]ince the map.el version header still shows 3.3.1, users think they have map v3.3.1 installed, same as the one available from GNU ELPA, while in fact they have map.el from Emacs master ("30.0.50") which will probably be tagged as map v3.4 someday.

I agree this isn't great.
elpaca-info displays both the declared "Version" package header and the git commit to avoid that ambiguity. It's designed with debugging in mind and provides an overview of the package (recipe, installed version, logs, etc). If your users are using Elpaca, request they M-x elpaca-info package-ID and it should help debug issues. straight-bug-report-package-info will print similar information, but only in a test environment. Straight should have a similar straight-info command to show package details.

I'm currently struggling financially, so I won't have time to implement any of the above for now. Once I can I will.

[raxod502]

I'm currently struggling financially, so I won't have time to implement any of the above for now

Take care, I will continue triaging issues with the time that I have available, never feel obligated to spend time you don't have.

We could prevent this and similar issues if straight didn't install external versions of core packages unless they were explicitly requested by the user

Ugh... this is really making it sound like we may actually need to properly support package version numbers. Because we really do want to install an external version of a package when the version shipped with Emacs is insufficiently recent to satisfy a dependency. But of course then if that is determined after the built-in version was already loaded, we are stuck again...

I think you're right and disabling built-in packages by default is the right solution, even if it will be disruptive. We can implement it in a way that will signal a warning on first use, perhaps. Or make it as an opt-in feature, and report a warning when a package is added to load-path when it was already loaded, with a pointer to the user option.

I'll add the latter idea to my to-do list, to resolve that long-standing issue. And then Nicholas' idea about installing built-in packages only when requested.