WIP: Async Support

[tessi]

This is a work-in-progress to implement an async rust backend for wasmex. Have a look at the async_todo.md (and keep it up-to-date).

See: #394

When working on this, make sure to export WASMEX_BUILD=true (this let's you build the rust binary locally instead of downloading a prebuilt binary).

[tessi]

*looking at the merge conflicts - A rebase on the current main would probs be a good first step :)

[munjalpatel]

@tessi how are we doing with this PR?

What is remaining for us to get this to finish line?

Is there anything I can help with?
I am pretty familiar with Elixir.
Never done much Rust, but can figure things out if required.

[tessi]

hey @munjalpatel :) thanks for asking and reminding me. I'm afraid this PR got stuck half way (I got a baby in between - it's been a while - and turns out these babies eat your free time as if it is milk!). Now, wasmtime has seen a fair amount of churn over the versions and the way async is handled changed enough so that I'd probably restart the work on a fresh wasmtime version update.

Good news is, that I'm working on a wasmtime version update just because of that behind the scenes. I probably have a WIP PR ready for that version bump soon - if the baby let's me sleep enogh, that is :)

[munjalpatel]

@tessi congratulations on the baby!! Happy to help wherever I can :)

[tessi]

Hey @munjalpatel and @scrogson (maybe, hopefully! :D) I just pushed a WIP version of a new async attempt.
I'm landing at a Rust compilation error that I cannot fix on my own and need help.

the full error is listed below, but I also have a short video explaining what I think is the problem:
https://www.loom.com/share/de98a77891b64077b7129d67e5be20c2?sid=ad00fe3d-a247-4403-978c-04e87c1f6b85

I don't know a solution to this error as I understand it, but without solving it I'm afraid we cannot use async wasmtime :/

The error:

error: future cannot be sent between threads safely
   --> src/instance.rs:67:5
    |
67  | /     send_async_nif_result(env, async move {
68  | |         let mut store_or_caller = (store_or_caller_resource.inner.lock().map_err(|e| {
69  | |             format!("Could not unlock store resource as the mutex was poisoned: {e}")
70  | |         }))?;
...   |
100 | |         .map_err(|error| format!("{:?}", error))
101 | |     })
    | |______^ future created by async block is not `Send`
    |
    = help: within `{async block@src/instance.rs:67:32: 101:6}`, the trait `Send` is not implemented for `std::sync::MutexGuard<'_, StoreOrCaller>`, which is required by `{async block@src/instance.rs:67:32: 101:6}: Send`
note: future is not `Send` as this value is used across an await
   --> src/instance.rs:88:14
    |
68  |         let mut store_or_caller = (store_or_caller_resource.inner.lock().map_err(|e| {
    |             ------------------- has type `std::sync::MutexGuard<'_, StoreOrCaller>` which is not `Send`
...
88  |             .await
    |              ^^^^^ await occurs here, with `mut store_or_caller` maybe used later
note: required by a bound in `send_async_nif_result`
   --> src/task.rs:31:50
    |
27  | pub fn send_async_nif_result<T, E, Fut>(env: Env, future: Fut) -> NifResult<(Atom, Atom)>
    |        --------------------- required by a bound in this function
...
31  |     Fut: future::Future<Output = Result<T, E>> + Send + 'static,
    |                                                  ^^^^ required by this bound in `send_async_nif_result`

(please not that this is WIP and probably all tests fail, but I only care about fixing this error at the moment, the rest will follow)

[munjalpatel]

Hey @tessi thanks a lot for getting this going. As always, awesome work! The explainer video was very helpful -- unfortunately, my Rust is very rusty!! I hope @scrogson can point us in a right direction :)

[scrogson]

@tessi I'll try to pull in the code locally tonight and see if I can figure something out.