Can the relative paths for requiring Lua files be used in pre-render scripts too?

[jimjam-slam]

Description

Hey folks,

I understand that, as of Quarto 1.4, I ought to be able to call a secondary Lua file using a relative path. I'm able to make that work with a filter, but I'm hoping to reuse that code with a pre-render script that my extension also contributes as metadata.

To reproduce

quarto create project
quarto add jimjam-slam/test-quarto-lua-modules     # the extension itself is called my-extension

The extension has three Lua files:

• util.lua is my utility function, used by both the filter and the prerender script. It exports two functions, first_func() and second_func() (which each return a string)
• my-filter.lua is the filter, which imports util.lua
• my-prerender.lua is the pre-render script, which also imports util.lua

The extension contributes the filter, as well as metadata that triggers the prerender script.

The default file refers to the extension name under filters:

---
title: "Hello"
filters:
  - my-extension
---

If I comment out the last three lines of _extension.yml (disabling the prerender script), everything runs fine on render:

pandoc 
  to: html
  output-file: Hello.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  variables: {}
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Hello
  
In filter
/Users/jimjamslam/Code/tests/extension-doc/test-project
true    exit    0
Hello first_util from filter!
Output created: Hello.html

But when the pre-render script is enabled, it doesn't seem to be able to find the module:

In pre-render
/Users/jimjamslam/Code/tests/extension-doc/test-project
true    exit    0
Error running filter /Users/jimjamslam/Code/tests/extension-doc/test-project/_extensions/jimjam-slam/my-extension/my-prerender.lua:
...ct/_extensions/jimjam-slam/my-extension/my-prerender.lua:4: module './util' not found:
        no field package.preload['./util']
        no file '/usr/local/share/lua/5.4///util.lua'
        no file '/usr/local/share/lua/5.4///util/init.lua'
        no file '/usr/local/lib/lua/5.4///util.lua'
        no file '/usr/local/lib/lua/5.4///util/init.lua'
        no file './//util.lua'
        no file './//util/init.lua'
        no file '/usr/local/lib/lua/5.4///util.so'
        no file '/usr/local/lib/lua/5.4/loadall.so'
        no file './//util.so'
        no file '/usr/local/lib/lua/5.4/.so'
        no file '/usr/local/lib/lua/5.4/loadall.so'
        no file './.so'
stack traceback:
        ...ct/_extensions/jimjam-slam/my-extension/my-prerender.lua:4: in main chunk
ERROR: Error
    at runScripts (file:///Users/jimjamslam/code/tools/quarto-cli/src/command/render/project.ts:993:15)
    at eventLoopTick (ext:core/01_core.js:175:7)
    at async runPreRender (file:///Users/jimjamslam/code/tools/quarto-cli/src/command/render/project.ts:940:3)
    at async renderProject (file:///Users/jimjamslam/code/tools/quarto-cli/src/command/render/project.ts:357:5)
    at async Command.actionHandler (file:///Users/jimjamslam/code/tools/quarto-cli/src/command/render/cmd.ts:251:26)
    at async Command.execute (https://deno.land/x/[email protected]/command/command.ts:1948:7)
    at async Command.parseCommand (https://deno.land/x/[email protected]/command/command.ts:1780:14)
    at async quarto (file:///Users/jimjamslam/code/tools/quarto-cli/src/quarto.ts:191:5)
    at async file:///Users/jimjamslam/code/tools/quarto-cli/src/quarto.ts:223:5
    at async file:///Users/jimjamslam/code/tools/quarto-cli/src/core/main.ts:45:14

If I change he import in the prerender script to refer down through the extension folder then I'm fine:

util = require("_extensions.jimjam-slam.my-extension.util")

In pre-render
/Users/jimjamslam/Code/tests/extension-doc/test-project
true    exit    0
Hello first_util from pre-render!


pandoc 
  to: html
  output-file: Hello.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  variables: {}
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Hello
  
In filter
/Users/jimjamslam/Code/tests/extension-doc/test-project
true    exit    0
Hello first_util from filter!
Output created: Hello.html

But I don't want to hardcode the extension path, since it can be installed namespaced by org or not. I also tried this:

util = require("_extensions.jimjam-slam.my-extension.util;_extensions.my-extension.util")

But this only works if the path that matches the reprex's extension path is last (I was hoping it would try both after reading the Lua docs).

Is there a way to reliably ensure that modules are found from pre-render scripts? I was also thinking about some sort of try-catch pattern, but this feels like overengineering!

Quarto check

Quarto 99.9.9
[✓] Checking environment information...
      Quarto cache location: /Users/jimjamslam/Library/Caches/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.6.3: OK
      Dart Sass version 1.85.1: OK
      Deno version 1.46.3: OK
      Typst version 0.13.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 99.9.9
      commit: 6202430787bc0787b97c19026cb96aa4e8a3ea19
      Path: /Users/jimjamslam/code/tools/quarto-cli/package/dist/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Tex:  (not detected)

[✓] Checking Chrome Headless....................OK
      Using: Chrome found on system
      Path: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
      Source: MacOS known location

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.13.3
      Path: /opt/homebrew/opt/[email protected]/bin/python3.13
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.4.1
      Path: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources
      LibPaths:
        - /Users/jimjamslam/Library/R/arm64/4.4/library
        - /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
      knitr: 1.49
      rmarkdown: 2.29

[✓] Checking Knitr engine render......OK

[jimjam-slam]

Here's a bit of a brute force method that checks both possible extension paths (for the pre-render script only, not the filter). It works, though!

-- try to load a module from multiple paths
-- modified version of https://stackoverflow.com/a/17878208/3246758
local function try_require(path_table)
  for _, v in pairs(path_table) do
    ok, err = pcall(require, v)
    if ok then
      return err
    end
  end
  error("Could not find a module to load from selected paths")
end

util = try_require({
  "_extensions.jimjam-slam.my-extension.util",
  "_extensions.my-extension.util"
})

[mcanouil]

I believe this is the current solution until the extension mechanism gets some rework/enhancements.

[mcanouil]

An alternative could be to use quarto inspect (Quarto CLI >= 1.8.5) as it now reports the extensions installed including their paths.

• Report extensions in quarto inspect #12733

[jimjam-slam]

Makes sense! Thanks, @mcanouil!

[cscheid]

I created #12913 to track.