Attempting to compile for DuckDB-Wasm, error when executing functions

[carstonhernke]

Hi! First of all, fantastic work on this library. I've been using it a bit with DuckDB locally and it's super helpful for many analysis tasks.

Now I'm going a step further and attempting to build this library for the DuckDB Wasm distribution. After a bit of fiddling around with various build settings, I was successfully able to build a h3ext.duckdb_extension.wasm file which I can load into a Wasm instance of DuckDB with load 'h3ext.duckdb_extension.wasm'. Great!

Now I'm running into an issue when I try to call functions from this library. When I call any of the functions, I get a rather cryptic error message. For example:

TypeError: Cannot read properties of undefined (reading 'apply') at wasmImports.<computed>.stub._.<computed> (duckdb-browser-eh.worker.c067f4b4d5eff325690c.js:2:17523)

The interesting thing is that duckdb seems to have registered the functions correctly, because if I call them with the wrong types, it gives me a helpful error message:

So it seems that something is going on when the functions actually execute, or when results are returned.

I have no idea how to even begin to debug this... any ideas? Thanks!

[carlopi]

I just bumped (and hopefully properly solved) this problem, I think there is something wrong with how target_link_libraries (https://github.com/isaacbrodsky/h3-duckdb/blob/main/CMakeLists.txt#L64C1-L64C22) is handled for Wasm extensions build process. I will update here in a bit, there is both a workaround (that I need to double-check) and a proper fix to CMake logic I need to PR.

[isaacbrodsky]

Ok, looking forward to that solution. In the mean time if you could post the build steps you are using that would be helpful for me to review the PR.

There is another option if the linking cannot be made to work which is the compile the H3 core library directly into the extension (e.g. reference its source files), although it is not preferable for maintainability reasons.

[carstonhernke]

@isaacbrodsky I'm using the Makefile in the DuckDB Wasm library for the build:

1. Edit .github/config/extension_config_wasm.cmake to include the following (I'm using a local source directory):

duckdb_extension_load(h3_ext
    DONT_LINK
    SOURCE_DIR /Users/XXXX/documents/git/h3-duckdb
)

2. Run make build_loadable_unsigned
3. Run make serve_loadable

@carlopi what is the workaround you mentioned?

[carlopi]

First, it can work:

What is missing is that the external libraries are not properly linked together, I have not a proper fix yet, but the workaround is doing:

make build_loadable_unsigned
ls -la **/h3ext.duckdb_extension
cd path/to/where/h3ext.duckdb_extension/is
emcc h3ext.duckdb_extension h3/lib/libh3.a -sSIDE_MODULE=1 -o h3ext.duckdb_extension.wasm -O3
file h3ext.duckdb_extension.wasm

And this is the file, if you want to give it a go (not signed and to be ungzipped):
h3ext.duckdb_extension.wasm.gz

[carlopi]

I found also another problems with this, that is in duckdb currently the build_loadable_extension CMake call assumes no other submodule but DuckDB, and this should be changed to "all but duckdb".
Solving this would allow to enable building using the GIT_URL syntax without requiring the local git clone (and there submodules are fetched by hand).

H3-DuckDB side I found some issue wiht ENABLE_LINTING being on by default, since on cross-compilation this adds some issue, but once found it can be worked around.

[isaacbrodsky]

I was not able to get h3-duckdb to execute in duckdb-wasm in my test this evening.

Here are my notes:

First, you must install the following additional dependencies
https://emscripten.org/docs/getting_started/downloads.html#installation-instructions-using-the-emsdk-recommended
https://ccache.dev/
A recent version of Rust
A recent version of Bash through Brew

Tested on duckdb-wasm revision f8e1b1ef19d1a66fd23bb4c485607b1d66f9f49b

Add the following to .github/config/extension_config_wasm.cmake

duckdb_extension_load(h3_ext
    DONT_LINK
    SOURCE_DIR /replace/with/path/to/h3-duckdb
)

# from duckdb-wasm
source ../emsdk/emsdk_env.sh
git submodule update --init --recursive
make build_loadable_unsigned
ls -la **/h3ext.duckdb_extension
# NOTE: The following used a different name for the extension directory!
cd  build/relsize/eh/third_party/duckdb/src/duckdb_ep-build/extension/h3_ext/
emcc h3ext.duckdb_extension h3/lib/libh3.a -sSIDE_MODULE=1 -o h3ext.duckdb_extension.wasm -O3
file h3ext.duckdb_extension.wasm
cd ../../../../../../../../../
make serve_loadable
# fails...
make
# fails...
make serve_loadable
# cancelled somewhere in the headless tests
make serve_loadable
# fails on no http-server
yarn add -D http-server -W
yarn run http-server packages/duckdb-wasm-app/build/release -o

Unfortunately the following command, shown above, does not work:

load 'h3ext.duckdb_extension.wasm';

Transcript:

duckdb> load 'h3ext';
IO Error: Extension "/home/web_user/.duckdb/extensions/v0.9.2/wasm_eh/h3ext.duckdb_extension" not found.

Candidate extensions: "s3", "http", "inet"

duckdb> load 'h3ext.duckdb_extension.wasm';
IO Error: Extension "h3ext.duckdb_extension.wasm" not found.

Candidate extensions: "excel"

Side note: https://duckdb.org/2023/12/18/duckdb-extensions-in-wasm.html suggests that the extension template creates loadable extensions, and from quickly looking at its Makefile it seems to be doing something similar to what Carston mentioned.

I subsequently tried an http URL:

make app_server

duckdb> load 'http://127.0.0.1:9003/duckdb-wasm/3c695d7ba9/wasm_eh/h3ext.duckdb_extension.wasm';
IO Error: Extension "http://127.0.0.1:9003/duckdb-wasm/3c695d7ba9/wasm_eh/h3ext.duckdb_extension.wasm" could not be loaded because its signature is either missing or invalid and unsigned extensions are disabled by configuration (allow_unsigned_extensions)

allow_unsigned_extensions is indeed off according to DuckDB's settings. Perhaps it got overwritten when I ran make earlier because of some other transient build failure.

I then tried:

make clean
make build_loadable_unsigned
ls -la **/h3ext.duckdb_extension
cd  build/relsize/eh/third_party/duckdb/src/duckdb_ep-build/extension/h3ext/
emcc h3ext.duckdb_extension h3/lib/libh3.a -sSIDE_MODULE=1 -o h3ext.duckdb_extension.wasm -O3
file h3ext.duckdb_extension.wasm
cd ../../../../../../../../../
make serve_loadable
make app_server

duckdb> load 'http://127.0.0.1:9003/duckdb-wasm/3c695d7ba9/wasm_eh/h3ext.duckdb_extension.wasm';
┌┐
└┘

duckdb> select h3_latlng_to_cell(0,0,0);
t is undefined

This error message is very cryptic and it is not clear to me how to continue debugging. Perhaps this is an issue with duckdb-wasm HEAD?

[isaacbrodsky]

I tried this again today and it doesn't seem to be in any better a situation. I think the extension template is building the WASM module correctly, since there have been an number of upstream updates there. I could not get duckdb-wasm to load it though (I seemed to get stuck in a situation where duckdb-wasm would not longer give me a prompt in the browser) If someone else with a working duckdb-wasm setup could try with the wasm binaries built by make wasm_* here I would appreciate it.

[carlopi]

Sent a PR with updates: #90

[jcolot]

@carstonhernke It seems to work now, here's an example: Observable

[carlopi]

Hi! responding also here, DuckDB-Wasm build process has been improved in the upcoming v0.10.2.

For v0.10.1 I would recommend looking at the Makefile, there are some extra steps like https://github.com/isaacbrodsky/h3-duckdb/blob/main/Makefile#L133 that are basically doing an extra (and external) linking step between the h3 library and the compiled wasm extension.

I would recommend just using the Makefile / mimicking CI.

[carstonhernke]

Thank you @carlopi !