ENH support external dependencies in widget development

[Charlie-XIAO]

Warning

We are likely going to use rolldown to replace SWC which will hopefully make bundling external dependencies more elegant, by which this PR will be superceded.

Earlier comments until 5a06da6

This PR aims to add support for external dependencies.

Testing this PR

Remove all node_modules/ including those in subdirectories, if you have not completed the migration towards pnpm in #141.

pnpm install
cd tooling/apis/ && pnpm build && cd ../../
cd tooling/react/ && pnpm build && cd ../../
cd tooling/ui/ && pnpm build && cd ../../
pnpm tauri dev  # or: pnpm tauri build --debug

I prepared a calendar scheduler widget which @ROMEEZHOU initially proposed: scheduler.zip. It uses some date-related utilities in date-fns and some icons in @radix-ui/react-icons. You can download this zip and unzip in the widget folder, then first act as a widget user and render directly. Since it ships the __external_dependencies.js file with it (which is the auto-generated bundle of external dependencies that we will mention later), a normal widget user can use it directly without having node or npm available. Then try removing the __external_dependencies.js file (not necessary, but just to make things clearer) and act as a widget developer. In this role node and npm are required, and first npm install for that widget. Then click the Bundle button in the corresponding widget tab: it will start loading and after it completes (hopefully succeeds), re-render the widget and it should work as expected.

Bundling external dependencies

The core problem here is: how to deal with node modules. I tried the SWC NodeModulesResolver, but it implements only a very simplistic node modules resolution algorithm. There are too many corner cases when it comes to package.json. For instance, simply locating the entry file is difficult, which can be specified in the browser key, in the exports configuration, etc.

The workaround I came up with in this PR is to partially rely on rollup for this purpose. Observe that node modules are not shipped with the widgets so they only come from node and npm. Indeed, only widget developers need this functionality, either creating a widget or modifying and existing one. Widget users never need to do this and instead relies on the bundle of external dependencies __external_bundle.js shipped with the widget. The good thing about rollup is that it has plugins @rollup/plugin-node-resolve and @rollup/plugin-commonjs that can help us handle the complex frontend ecosystem and the tons of ways of saying the same thing, which is clearly not our expertise and not maintainable in this project. The workflow is approximately as follow:

I. Produce the bridge module

"Bridge module" here refers to a module that (1) conveys what needs to be imported and (2) provides named exports for everything needed. Let's say the widget source code contains

import a, { b as c, d } from "e";
import * as f from "g";

We should produce a bridge module like

import a, { b as c, d } from "e";
import * as f from "g";
export { a, c, d, f };

It takes the import statements as is and exports their local representation so that all these imports can later be accessed from one single bundle. This is done via an AST transform (see ExternalImportInspector in src-tauri/src/bundler/transforms.rs). It is applied on the bundled AST of the widget and is actually an inspector rather than a transformer, in that it only records information from but not alter the AST itself.

Note the importance of applying ExternalImportInspector on the bundled AST instead of on the AST of each source file. For instance, say we have the same statement import a from "mod" in two source files. In the bundle, the local variable name needs to be distinguished and thus we will see something like import a from "mod" and import a1 from "mod". If ExternalImportInspector is applied per file AST, we will have no idea what a1 refers to.

II. Convert the bridge module into the external bundle

For this step we rely on rollup as mentioned. We first needs to install rollup and the following plugins as devDependencies:

• @rollup/plugin-alias: Redirect react to @deskulpt-test/react so as to reuse the react at runtime.
• @rollup/plugin-replace: Replace process.env.NODE_ENV with "production". Some external packages tell production or development build based on this variable, but process is unavailable in browsers and we know we want the production build so there is no need to include the development-specific code.
• @rollup/plugin-commonjs: Convert CommonJS modules to ES6 which we require.
• @rollup/plugin-node-resolve Resolve node modules.
• @rollup/plugin-terser: Uglify to produce minimal bundles.

Then execute the following command:

npx rollup __external_bundle_bridge.js \
    --file __external_bundle.js \
    --format esm \
    --external @deskulpt-test/react \
    --plugin "alias={entries:{react:'@deskulpt-test/react'}}" \
    --plugin "replace={'process.env.NODE_ENV':JSON.stringify('production'),preventAssignment:true}" \
    --plugin commonjs \
    --plugin node-resolve \
    --plugin terser

We use purely command line instead of writing a configuration file because it's perhaps faster and safer. This will produce __external_bundle.js which resolves all those imports and exports everything needed by the bundled widget AST. Then delete the bridge file.

Bundling widget source code

We follow essentially the same logic as before when there are no external dependencies, and perform an additional round of bundling to resolve the imports of external dependencies when there are. The workflow is approximately as follows:

I. Produce the bundle without resolving external dependencies

After bundling into AST and applying TypeScript and JSX transforms (same logic as before), if there are no external dependencies, we simply rename APIs into the corresponding blob URL and emit the code. This is just completely the same as what we have on main. Otherwise (i.e., there are external dependencies), we need to apply a transform to redirect the imports (see ExternalImportRedirector in src-tauri/src/bundler/transforms.rs). Using the same example, say we have

import a, { b as c, d } from "e";
import * as f from "g";

They will be converted into

import { a, c, d } from "${ABS_PATH_TO_ROOT}/__external_bundle.js";
import { f } from "${ABS_PATH_TO_ROOT}/__external_bundle.js";

It is clear that this corresponds to export statements in the bridge module mentioned above, and thus corresponds to the export statements in __external_bundle.js as well. We then emit the module into a temporary file __temp_widget_bundle.js. Note that we are not renaming APIs yet in this case because of the need to bundle a second time, as we will specify in the following step.

II. Resolve external dependencies

The previous step only redirects the external imports but not actually bundle them together with the widget source code. It also does not work directly because Tauri limits access to local files. It is quite tricky how to play with the allowlist scope etc., in particular the security part of Tauri v2 is not yet well-documented (and perhaps not very stable either), and it is hard to come up with a proper pattern for the allowlist scope. My choice in this PR is thus to bundle a second time.

We follow the mostly the same logic as the firsts round of bundling. The only import statements left should be those to __external_bundle.js, imports of default dependencies, and imports from @deskulpt-test/emotion/jsx-runtime.js. Then we apply the transform to rename APIs.

Asynchronous command

See Tauri documentation on async commands. My initial implementation used a synchronous command, and when invoking the command I used tauri::async_runtime::block_on. This will simply cause all other functionalities and even the UI to freeze since it is essentially blocking on the main thread. Async command is much better in this case regarding performance as it will be invoked on a separate thread. In fact, I just realized that all commands should theoretically be made asynchronous unless there is good reason not to do so, since noticeable or not it will give better theoretical performance but these can be tackled in a dedicated PR in the future. For more information, you may watch this YouTube video made by one of the Tauri maintainers.

[github-actions]

✔️ Deskulpt Built Successfully!

Deskulpt binaries have been built successfully on all supported platforms. Your pull request is in excellent shape! You may check the built Deskulpt binaries here and download them to test locally.

_{Workflow file: .github/workflows/build.yaml. Generated for commit: 0242381.}