tree-sitter tsc module will be re-built instead of being downloaded with straight-rebuild-package

[scturtle]

What's wrong

Tree-sitter tsc module will be re-built instead of being downloaded with straight-rebuild-package.

Directions to reproduce.

.emacs.d/init.el:

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'use-package)
(setq straight-use-package-by-default t)

(use-package tree-sitter)
(use-package tree-sitter-langs)

Run emacs --init-directory /tmp/.emacs.d the first time, the tsc module should be downloaded correctly.

In Emacs, run C-u M-x straight-rebuild-package and enter tree-sitter to rebuild tree-sitter and tsc. The tsc module will be built from source this time.

details

*straight-byte-compilation*:

In toplevel form:
tree-sitter-query.el:15:2: Error: Module is not GPL compatible: "/tmp/.emacs.d/straight/build/tree-sitter-langs/bin/scheme.dylib"
Compiling /tmp/.emacs.d/straight/build/tree-sitter/tree-sitter.el...
Done (Total of 6 files compiled, 1 failed, 1 skipped)

it seems that .emacs.d/straight/build/tree-sitter-langs/bin is included in load-path while doing straight-rebuild-package. Then during downloading tsc module with url-copy-file, not knowing why, Emacs will try to load the tree-sitter language files *.dylib / *.so as Emacs module and fail. Then tree-sitter will turn to compile tsc from source.

Version information

• Emacs version: 29.1
• Operating system: MacOS and Linux

[raxod502]

I think you're right. Furthermore I think the reason the subdirectory is (incorrectly) added to load-path is this:

straight.el/straight.el

Line 5270 in e0f3ba8

(normal-top-level-add-subdirs-to-load-path)

It seems that as a convenience we add all subdirectories in ~/.emacs.d/straight/build to load-path, which isn't right. We should only be adding the packages that are dependencies of the one being built; furthermore, even amongst those, we shouldn't be adding subdirectories recursively, only the top level ones.

[progfolio]

I think you're right. Furthermore I think the reason the subdirectory is
(incorrectly) added to load-path is this:

straight.el/straight.el

Line 5270 in e0f3ba8

(normal-top-level-add-subdirs-to-load-path)

It seems that as a convenience we add all subdirectories in
~/.emacs.d/straight/build to load-path, which isn't right.

This was a compromise added in #812.
Notes from 01eb849:

Previous method involved looking up each package's dependencies.
It did not account for local-repo names and transitive dependencies.

I can't recall why the :local-repo keyword was causing issues (if it was at all).
It looks like straight--symlink-package uses (straight--build-dir package), so I don't think it should be a problem.

We should only be
adding the packages that are dependencies of the one being built; furthermore,
even amongst those, we shouldn't be adding subdirectories recursively, only the
top level ones.

Agreed. Now that we have straight-dependencies we can easily get the transitive dependencies (it looks like my initial attempt relied on straight--get-dependencies, which did not return transitive dependencies, leading to the compromise).

#1132 seems to work for the case in this issue (see below), but it could use some more testing.

Test Case

(straight-bug-report
  :pre-bootstrap
  (setq straight-repository-user "progfolio" straight-repository-branch
        "fix/compile")
  :post-bootstrap
  (straight-use-package 'tree-sitter)
  (straight-use-package 'tree-sitter-langs)
  (straight-rebuild-package "tree-sitter" 'recusrive)
  (with-current-buffer (get-buffer-create "*straight-byte-compilation*")
    (message "%S" (buffer-substring-no-properties (point-min) (point-max)))))

• Test run at: 2023-11-17 23:34:40
• system-type: gnu/linux
• straight-version: prerelease (HEAD -> fix/compile, fork/fix/compile) 058bcdd 2023-11-17
• emacs-version: GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38, cairo version 1.18.0) of 2023-11-16

Output

Bootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> fix/compile, origin/fix/compile) 058bcdd 2023-11-17
Cloning elisp-tree-sitter (for tree-sitter)...
Cloning elisp-tree-sitter (for tree-sitter)...done
Building tree-sitter...
Building tree-sitter → Building tsc...
Building tree-sitter → Building tsc...done
Building tree-sitter...
Building tree-sitter...done
Cloning tree-sitter-langs...
Cloning tree-sitter-langs...done
Building tree-sitter-langs...
Building tree-sitter-langs...done

"
$ /usr/bin/emacs -Q --batch --eval \\
 \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/straight/\\\") 'nil load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/straight/\\\" 0 'force))\"
Checking /tmp/straight.el-test-86hhVW/straight/build/straight/...
Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-autoloads.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-ert-print-hack.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight-x.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/straight/straight.el...
Done (Total of 3 files compiled, 1 skipped)

$ /usr/bin/emacs -Q --batch --eval \\
 \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\") 'nil load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\" 0 'force))\"
Checking /tmp/straight.el-test-86hhVW/straight/build/tsc/...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-autoloads.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-dyn-get.el...

In tsc-dyn-get--build-version:
tsc-dyn-get.el:185:2: Warning: docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...')
Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc-obsolete.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tsc/tsc.el...
tsc-dyn-get: Using source :github (:loaded nil :recorded nil :requested 0.18.0)
tsc-dyn-get: Downloading https://github.com/emacs-tree-sitter/elisp-tree-sitter/releases/download/0.18.0/tsc-dyn.x86_64-unknown-linux-gnu.so
Contacting host: github.com:443
Checking /tmp/straight.el-test-86hhVW/straight/build/tsc/src...
Done (Total of 3 files compiled, 1 skipped)

$ /usr/bin/emacs -Q --batch --eval \\
 \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\") '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\") load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\" 0 'force))\"
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-autoloads.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-cli.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-debug.el...
tsc-dyn-get: Using source :github (:loaded nil :recorded 0.18.0 :requested 0.18.0)
tsc-dyn-get: Recorded version already satifies requested -> loading
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-extras.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-hl.el...

In toplevel form:
tree-sitter-hl.el:263:12: Warning: custom-declare-variable ‘tree-sitter-hl-face-mapping-function’ docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...')

In tree-sitter-hl--highlight-region:
tree-sitter-hl.el:487:12: Warning: ‘inhibit-point-motion-hooks’ is an obsolete variable (as of 25.1); use ‘cursor-intangible-mode’ or ‘cursor-sensor-mode’ instead

In tree-sitter-hl-mode:
tree-sitter-hl.el:639:2: Warning: docstring has wrong usage of unescaped single quotes (use \\=' or different quoting such as `...')
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-load.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter-query.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter/tree-sitter.el...
Done (Total of 7 files compiled, 1 skipped)

$ /usr/bin/emacs -Q --batch --eval \\
 \"(progn (setq load-path (append '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/\\\") '(\\\"/tmp/straight.el-test-86hhVW/straight/build/tsc/\\\" \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter/\\\") load-path)) (byte-recompile-directory \\\"/tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/\\\" 0 'force))\"
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs-autoloads.el...
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs-build.el...

In tree-sitter-langs-install-grammars:
tree-sitter-langs-build.el:500:22: Warning: ‘unless’ with empty body

In tree-sitter-langs-install-latest-grammar:
tree-sitter-langs-build.el:550:2: Warning: docstring wider than 80 characters
Compiling /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/tree-sitter-langs.el...
tsc-dyn-get: Using source :github (:loaded nil :recorded 0.18.0 :requested 0.18.0)
tsc-dyn-get: Recorded version already satifies requested -> loading
tree-sitter-langs: Installing grammar bundle v0.12.79 (was vnil)
Contacting host: github.com:443
[tree-sitter-langs] Running (tar -xvzf tree-sitter-grammars.x86_64-unknown-linux-gnu.v0.12.79.tar.gz) in /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/bin/


In tree-sitter-langs--hl-default-patterns:
tree-sitter-langs.el:245:28: Warning: ‘ignore-error’ condition argument should not be quoted: 'file-missing
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/agda...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/arduino...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/asm...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/astro...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/bash...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/bibtex...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/c...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/clojure...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/cmake...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/cpp...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/css...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/csv...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/dart...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/dockerfile...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/elixir...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/elm...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/fish...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gdscript...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/git-rebase...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitattributes...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitcommit...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/gitignore...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/glsl...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/go...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/groovy...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/haskell...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/hcl...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/heex...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/hlsl...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/html...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/jai...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/java...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/javascript...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/json...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/kotlin...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/lua...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/make...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/matlab...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/mermaid...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ocaml...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ocaml-interface...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/pascal...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/pgn...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/php...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/psv...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/python...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/r...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/ruby...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/rust...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/solidity...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/sql...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tcl...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tsv...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/tsx...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/twig...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/typescript...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/typst...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/xml...
Checking /tmp/straight.el-test-86hhVW/straight/build/tree-sitter-langs/queries/yaml...
Done (Total of 2 files compiled, 1 skipped)
BUNDLE-VERSION
ada.so
agda.so
arduino.so
asm.so
astro.so
bash.so
beancount.so
bibtex.so
c-sharp.so
c.so
clojure.so
cmake.so
commonlisp.so
cpp.so
css.so
csv.so
d.so
dart.so
dockerfile.so
dtd.so
elisp.so
elixir.so
elm.so
erlang.so
fennel.so
fish.so
fluent.so
fortran.so
gdscript.so
git-rebase.so
gitattributes.so
gitcommit.so
gitignore.so
gleam.so
glsl.so
go.so
groovy.so
haskell.so
hcl.so
heex.so
hlsl.so
html.so
jai.so
janet-simple.so
java.so
javascript.so
jsdoc.so
json.so
jsonnet.so
julia.so
kotlin.so
latex.so
lua.so
make.so
markdown.so
matlab.so
mermaid.so
meson.so
nix.so
noir.so
ocaml-interface.so
ocaml.so
org.so
pascal.so
perl.so
pgn.so
php.so
prisma.so
python.so
r.so
rst.so
ruby.so
rust.so
scala.so
scheme.so
smithy.so
solidity.so
sql.so
swift.so
tcl.so
toml.so
tsx.so
twig.so
typescript.so
typst.so
verilog.so
vhdl.so
xml.so
yaml.so
zig.so
"
Packages:
"straight"                n/a                  fix/compile 058bcdd 2023-11-17
"org-elpa"                n/a                  n/a
"melpa"                   n/a                  master d01c726f 2023-11-12
"gnu-elpa-mirror"         n/a                  master 3e9da80 2023-11-08
"nongnu-elpa"             n/a                  main e9d06c3 2023-11-07
"el-get"                  melpa                master a5ba5852 2023-10-20
"emacsmirror-mirror"      n/a                  master 5cbbb0a 2023-11-13
"tree-sitter"             melpa                release 3cfab8a 2022-02-13
"tsc"                     melpa                release 3cfab8a 2022-02-13
"tree-sitter-langs"       melpa                release 3a3ad05 2023-11-15

[raxod502]

Excellent, thank you so much! I was not looking forward to that refactor.