Welcome to libsql Discussions!

[glommer]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[xfbs]

Hey @glommer — awesome idea you guys are working on. I think having some support for WASM in SQLite would be incredible. I imagine a future where you can write things like CHECK constraints in any language you please, have them compile to WASM and send that as a blob to SQLite as part of a migration. Example:

#[libsql::constraint]
fn check_is_ipv4(data: &[u8]) -> Result<(), ConstraintError> {
    Ipv4Addr::try_from(&data)?;
    Ok(())
}

I feel like this feature alone is worth it's weight in gold, because it means you can "extend" SQLite more easily with custom data types.

The other idea you guys have discussed — bringing io_uring support into SQLite and making an async interface — is likewise a really awesome idea. I'm a bit worried, tho, that this might be a lot of work, essentially requiring reworking the entire SQLite API?

In any case, I hope this project succeeds. What's the next steps?

[glommer]

Thanks @xfbs!

Full disclosure: agree on io_uring, and although this is the stuff I know the most from previous projects, I'll likely defer until we gain more steam. The ideal scenario may require a different API, and I don't think we earned the right yet to ask that of people (to be clear, we'd not replace the API, we'd add one so things keep working).

My friend and colleague @psarna is looking into WASM stuff, since that's his forte. I am tagging him here so you can discuss specifics.

Next steps depend on how much you want to get involved!

[xfbs]

I’d be down to help figure out things! I have a pretty strong C and Rust background, so I could be useful. Tho I am not familiar with the SQLite codebase apart from as an end user so far. But yeah, I feel like adding WASM capabilities is something that should be easily possible.

[psarna]

@xfbs that's an amazing idea, let me translate it to a github issue and we can continue the design and implementation in there.

[psarna]

@xfbs and as for your io_uring question - as explained in #3, I think sqlite is not that far from being able to handle async I/O. It already has a poll-like interface where it returns SQLITE_BUSY as soon as it's not able to proceed, and expects the user to retry again later. sqlite3_stmt is already a state machine, so it sounds feasible to extend the state with tracking pending async I/O requests, and poll them each time a user wants to proceed. If the operation is ready, we can return the result, and if it's not the case yet, we just return SQLITE_BUSY or its equivalent.

[xfbs]

Makes sense. I would be happy to try to help with that as well — been reading a lot about io_uring lately, would be cool to get to build something with it.

[jschlesser]

I would like to see WASI as a compile target for libsql so that It could be compiled into WASI applications. This would potentially enable a heterogeneous mix of users and their sqlite backed applications to be hosted on the same OS safely, potentially even in the same OS process. Mix in a little LiteFS behind it and I'm sure everybody that ends up here can see where this leads. Note that this is a distinct concept from WASM support inside of libsql.

Anecdotally....
I've spent a few hours trying to compile sqlite using the wasi-sdk clang and it didn't 'just work'. There seems to be some lingering issues to be resolved, or maybe its just the right set of clang params to appease the WASI sysroot. I don't know if it's something that is far far away due to WASI features or really close and just needs a few fixes in libsql. I'm not a WASI expert nor sqlite expert, yet. I downloaded the recent sqlite amalgam and unzipped it and made a wasi directory in the same directory as the amalgam and downloaded and unpacked the wasi-sdk for macOS and the wasi-sysroot and unpacked them in the wasi directory.

I got as far as the following compile command.
./wasi/wasi-sdk-16.0/bin/clang -DSQLITE_THREADSAFE=0 -D_WASI_EMULATED_MMAN -D_WASI_EMULATED_SIGNAL --sysroot ./wasi/wasi-sysroot -Wall -o sqlite shell.c sqlite3.c

The errors left look like a bunch of stuff related to POSIX users and permissions and file locking etc...

[psarna]

For starters, I recommend this readme (from original sqlite) https://github.com/libsql/libsql/tree/main/ext/wasm - it explains the limitations, and in particular, look at the "no I/O" section. That limit is imposed because Wasm's first priority is isolation, and apis for accessing input/output aren't set in stone yet, they're just proposals, e.g.: https://github.com/WebAssembly/wasi-io

[xfbs]

Would it make sense to schedule a (weekly) meeting for this project? I feel like that might be a good way to get together, share some ideas and coordinate development.

[glommer]

@xfbs in general I don't think it makes sense to have scheduled meetings for OSS projects. If you do want to meet, I'm more than happy to schedule something! Best is to hang out on our discord or send messages here for day to day comms, and schedule meetings only with the interested parties as needed.

[xfbs]

@glommer I'd be down to meet! Do you think we can schedule something? I've joined the Discord as well and am tracking discussion that happens there. Would be nice to have an opportunity to sync up and get to know each other. :)