use-package integration causes incorrect symbol-file

[MasonProtter]

What's wrong

If I add a package with use-package and :straight t, then evaluating (symbol-file 'the-package) will give the location of the use-package call, not the location of the package itself. This is not the case when one adds the package with straight directly.

This has caused bugs in packages I use such as julia-snail

Directions to reproduce

Put the following in your init.el:

(use-package julia-snail
  :straight t)

and then evaluate (symbol-file 'julia-snail). This will return e.g. "/home/mason/.emacs.d/myinit.el", whereas if you do

(straight-use-package 'julia-snail)

then (symbol-file 'helm) correctly returns "/home/mason/.emacs.d/straight/build/julia-snail/julia-snail.elc"

Version information

• Emacs version: GNU Emacs 29.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.40, cairo version 1.18.0)
• Operating system: Manjaro Linux

[progfolio]

I get the same result with or without use-package on Emacs 30.0.50.

Test Case

(straight-bug-report
  :post-bootstrap
  (use-package julia-snail :defer t :straight t)
  (print (symbol-file 'julia-snail)))

• Test run at: 2024-02-29 12:22:29
• system-type: gnu/linux
• straight-version: prerelease (HEAD -> develop, origin/develop) b1062df 2024-02-26
• emacs-version: GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo version 1.18.0) of 2024-02-28

Output

Bootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) b1062df 2024-02-26
Cloning julia-snail...
Cloning julia-snail...done
Building julia-snail...
Building julia-snail → Cloning dash.el...
Building julia-snail → Cloning dash.el...done
Building julia-snail → Building dash...
Building julia-snail → Building dash...done
Building julia-snail → Cloning julia-emacs (for julia-mode)...
Building julia-snail → Cloning julia-emacs (for julia-mode)...done
Building julia-snail → Building julia-mode...
Building julia-snail → Building julia-mode...done
Building julia-snail → Cloning s.el...
Building julia-snail → Cloning s.el...done
Building julia-snail → Building s...
Building julia-snail → Building s...done
Building julia-snail → Cloning spinner...
Building julia-snail → Cloning spinner...done
Building julia-snail → Building spinner...
Building julia-snail → Building spinner...done
Building julia-snail → Cloning popup-el (for popup)...
Building julia-snail → Cloning popup-el (for popup)...done
Building julia-snail → Building popup...
Building julia-snail → Building popup...done
Building julia-snail...
Building julia-snail...done

"julia-snail"

Packages:
"straight"                n/a                  develop b1062df 2024-02-26
"org-elpa"                n/a                  n/a
"melpa"                   n/a                  master 0a85d1c8 2024-02-27
"gnu-elpa-mirror"         n/a                  master 06f5e40 2024-02-28
"nongnu-elpa"             n/a                  main 7c06709 2024-02-22
"el-get"                  melpa                master 4f0d43b3 2024-02-14
"emacsmirror-mirror"      n/a                  master cee023f 2024-02-20
"julia-snail"             melpa                master 4a0adf8 2024-02-27
"dash"                    melpa                master 5df7605 2024-02-16
"julia-mode"              melpa                master 7a8c868 2023-07-12
"s"                       melpa                master dda84d3 2023-04-14
"spinner"                 gnu-elpa-mirror      master 67681b8 2023-09-10
"popup"                   melpa                master 4d6f6c2 2024-01-19

Test Case

(straight-bug-report
  :post-bootstrap
  (straight-use-package 'julia-snail)
  (print (symbol-file 'julia-snail)))

• Test run at: 2024-02-29 12:21:46
• system-type: gnu/linux
• straight-version: prerelease (HEAD -> develop, origin/develop) b1062df 2024-02-26
• emacs-version: GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo version 1.18.0) of 2024-02-28

Output

Bootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) b1062df 2024-02-26
Cloning julia-snail...
Cloning julia-snail...done
Building julia-snail...
Building julia-snail → Cloning dash.el...
Building julia-snail → Cloning dash.el...done
Building julia-snail → Building dash...
Building julia-snail → Building dash...done
Building julia-snail → Cloning julia-emacs (for julia-mode)...
Building julia-snail → Cloning julia-emacs (for julia-mode)...done
Building julia-snail → Building julia-mode...
Building julia-snail → Building julia-mode...done
Building julia-snail → Cloning s.el...
Building julia-snail → Cloning s.el...done
Building julia-snail → Building s...
Building julia-snail → Building s...done
Building julia-snail → Cloning spinner...
Building julia-snail → Cloning spinner...done
Building julia-snail → Building spinner...
Building julia-snail → Building spinner...done
Building julia-snail → Cloning popup-el (for popup)...
Building julia-snail → Cloning popup-el (for popup)...done
Building julia-snail → Building popup...
Building julia-snail → Building popup...done
Building julia-snail...
Building julia-snail...done

"julia-snail"

Packages:
"straight"                n/a                  develop b1062df 2024-02-26
"org-elpa"                n/a                  n/a
"melpa"                   n/a                  master 0a85d1c8 2024-02-27
"gnu-elpa-mirror"         n/a                  master 06f5e40 2024-02-28
"nongnu-elpa"             n/a                  main 7c06709 2024-02-22
"el-get"                  melpa                master 4f0d43b3 2024-02-14
"emacsmirror-mirror"      n/a                  master cee023f 2024-02-20
"julia-snail"             melpa                master 4a0adf8 2024-02-27
"dash"                    melpa                master 5df7605 2024-02-16
"julia-mode"              melpa                master 7a8c868 2023-07-12
"s"                       melpa                master dda84d3 2023-04-14
"spinner"                 gnu-elpa-mirror      master 67681b8 2023-09-10
"popup"                   melpa                master 4d6f6c2 2024-01-19

Please try evaluating those test cases in your scratch buffer and sharing the output.

I'll note, too, that on my version of Emacs the docstring for symbol-file mentions:

If SYMBOLspecifies an autoloaded function, the value can be a relative file name without extension.

Which applies in this case. That may be a fragile solution for whatever julia-snail is trying to accomplish.

[raxod502]

I get this error when trying to load the package:

ELISP> (require 'julia-snail)
*** Eval error ***  Neither vterm nor eat dependencies detected; please install one or the other                                                                                              

However, modifying the init-file as follows can reproduce the problem:

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 6))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'vterm)
(straight-use-package 'julia-snail)

(require 'julia-snail)

As you can see there use-package does not have anything to do with the problem. Here are the test results:

ELISP> (featurep 'julia-snail)
t
ELISP> (symbol-file 'julia-snail)
"/home/raxod502/files/temp/jeorigjeorig/init.el"

Looking at load-history there is an entry for (defun . julia-snail) under julia-mode.elc, but there is also an entry for (autoload . julia-snail) under init.el. Because init.el finishes loading later, it is earlier on the list, so gets returned.

It was unclear to me how this was supposed to work so I set up package.el and installed julia-snail and vterm, then used this init-file:

(package-initialize)
(require 'julia-snail)

and this early init-file:

(setq package-enable-at-startup nil)

Turns out, the reason things work in this situation is that the autoloads are stored in an actual separate file, hence get added to load-history earlier:

 ("/home/raxod502/.emacs.d/elpa/julia-snail-20240227.1914/julia-snail-autoloads.el"
  (autoload . julia-snail)
  (autoload . julia-snail-mode)
  (autoload . julia-snail-repl-mode))

Whereas straight.el is just evaluating them. Easy fix - we have straight.el set the load file correctly while evaluating autoloads. I'll fix.

[raxod502]

Heh, well, we are already trying to do that. But apparently Emacs does not believe it.

straight.el/straight.el

Lines 5492 to 5498 in b1062df

	;; Some autoloads files expect to be loaded normally, rather
	;; than read and evaluated separately. Fool them.
	(let ((load-file-name (straight--autoloads-file package))
	(load-in-progress t))
	;; car is the feature list, cdr is the autoloads.
	(dolist (form (cdr (gethash package straight--autoloads-cache)))
	(eval form))))

It appears you can (setq straight-cache-autoloads nil) to work around the problem. Which means that something is different between the eval above, and the load that is the other branch:

straight.el/straight.el

Lines 5434 to 5435 in b1062df

	(when (file-exists-p autoloads-file)
	(load autoloads-file nil 'nomessage))))

[progfolio]

Radon Rosborough ***@***.***> writes:

Heh, well, we are already trying to do that. But apparently Emacs does not believe it. https://github.com/radian-software/straight.el/blob/b1062df10ba4c10ff7a3c61b9e124b3242b11bb2/straight.el#L5492-L5498 It appears you can (setq straight-cache-autoloads nil) to work around the problem. Which means that something is different between the eval above, and the load that is the other branch:

eval does not retain the outer lexical environment. We could either "bake" the form into the eval, or construct the lexical environment and pass it as the second arg to eval.

[raxod502]

It doesn't retain the lexical environment, but that is not an issue here because the variables we are binding are dynamic rather than lexical. It's an issue with what load does to load-history, which we didn't replicate. #1151 should fix this