Diplomat and UniFFI are both tools which expose a rust implemented API over an FFI. At face value, these tools are solving the exact same problem, but their approach is significantly different.
This document attempts to describe these different approaches and discuss the pros and cons of each. It's not going to try and declare one better than the other, but instead just note how they differ. If you are reading this hoping to find an answer to "what one should I use?", then that's easy - each tool currently supports a unique set of foreign language bindings, so the tool you should use is the one that supports the languages you care about!
Disclaimer: This document was written by one of the UniFFI developers, who has never used diplomat in anger. Please feel free to open PRs if anything here misrepresents diplomat.
See also: This document was discussed in this PR, which has some very interesting discussion - indeed, some of the content in this document has been copy-pasted from that discussion, but there's further detail there which might be of interest.
The key difference between these 2 tools is the "type system". While both are exposing Rust code (which obviously comes with its own type system), the foreign bindings need to know lots of details about all the types expressed by the tool.
For the sake of this document, we will use the term "type universe" to define the set of all types known by each of the tools. Both of these tools build their own "type universe" then use that to generate both Rust code and foreign bindings.
UniFFI's model is to parse an external ffi description from a .udl file which describes the
entire "type universe". This type universe is then used to generate both the Rust scaffolding
(on disk as a .rs file) and the foreign bindings.
What's good about this is that the entire type system is known when generating both the rust code and the foreign binding, and is known without parsing any Rust code. This is important because things like field names and types in structs must be known on both sides of the FFI.
What's bad about this is that the external UDL is very ugly and redundant in terms of the implemented rust API.
Diplomat defines its "type universe" (ie, the external ffi) using macros.
What's good about this is that an "ffi module" (and there may be many) defines the canonical API
and it is defined in terms of Rust types - the redundant UDL is removed.
The Rust scaffolding can also be generated by the macros, meaning there are no generated .rs
files involved. Types can be shared among any of the ffi modules defined in the project -
for example, this diplomat ffi module
uses types from a different ffi module.
Restricting the definition of the FFI to a single module instead of allowing that definition to appear in any Rust code in the crate also offers better control over the stability of the API, because where the FFI is defined is constrained. This is an explicit design decision of diplomat.
While the process for defining the type universe is different, the actual in-memory
representation of that type universe isn't radically different from UniFFI - for example,
here's the definition of a Rust struct,
and while it is built from a syn struct, the final representation is independent of syn
and its ast representation.
Ryan tried this same macro approach for UniFFI in #416 - but we struck a limitation in this approach for UniFFI's use-cases - the context in which the macro runs doesn't know about types defined outside of that macro, which are what we need to expose.
Let's look at diplomat's simple example:
#[diplomat::bridge]
mod ffi {
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}
impl MyFfiType {
pub fn create() -> MyFfiType { ... }
...
}
}This works fine, but starts to come unstuck if you want the types defined somewhere else. In this trivial example, something like:
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}
#[diplomat::bridge]
mod ffi {
impl MyFfiType {
pub fn create() -> MyFfiType { ... }
...
}
}fails - diplomat can't handle this scenario - in the same way and for the same reasons that Ryan's #416 can't - the contents of the struct aren't known.
From the Rust side of the world, this is probably solvable by sprinkling more macros around - eg, something like:
#[uniffi::magic]
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}might be enough for the generation of the Rust scaffolding - in UniFFI's case, all we really need
is an implementation of uniffi::RustBufferViaFfi which is easy to derive, and UniFFI can
generate code which assumes that exists much like it does now.
However, the problems are in the foreign bindings, because those foreign bindings do not know
the names and types of the struct elements without re-parsing every bit of Rust code with those
annotations. As discussed below, re-parsing this code might be an option if we help Uniffi to
find it, but asking UniFFI to parse this and all dependent crates to auto-discover them
probably is not going to be viable.
As mentioned above, diplomat considers the limitation described above as an intentional design
feature. By limiting where FFI types can be described, there's no risk of changes made "far away"
from the FFI to change the FFI. This was born of experience in tools like cbindgen.
For Uniffi, all use-cases needed by Mozilla don't share this design goal, primarily because the FFI is the primary consumer of the crate. The Rust API exists purely to service the FFI. It's not really possible to accidentally change the API, because every API change made will be in service of exposing that change over the FFI. The test suites written in the foreign languages are considered canonical.
In both diplomat and #416, the approach taken
is very similar - it takes a path to a the Rust source file/tree, and uses syn to locate the special modules (ie, ones annotated with #[diplomat:bridge] in the case of diplomat.)
While some details differ, this is just a matter of implementation - #416 isn't quite as aggressive about consuming the entire crate to find multiple FFI modules (and even then, diplomat doesn't actually process the entire crate, just modules tagged as a bridge), but could easily be extended to do so.
But in both cases, for our problematic example above, this process never sees the layout of the
MyFfiType struct because it's not inside the processed module, so that layout can't be
communicated to the foreign bindings.
As noted above, this is considered a feature for diplomat, but a limitation for UniFFI.
This is the problem which caused us to decide to stop working on #416 - the current world where the type universe is described externally doesn't have this limitation - only the UDL file needs to be parsed when generating the foreign bindings. The application-services team has concluded that none of our non-trivial use-cases for UniFFI could be described using macros, so supporting both mechanisms is pain for no gain.
As noted in #416, wasm-bindgen has a similarly shaped problem, and solves it by having
the Rust macro arrange for the resulting library to have an extra data section with the
serialized "type universe" - foreign binding generation would then read this information from the
already built binary. This sounds more complex than the UniFFI team has appetite for at
the current time.
Adapting the diplomat/#416 model to process the entire crate?
We noted that diplomat intentionally restricts where the ffi is generated, whereas UniFFI considers that a limitation - but what if we can teach UniFFI to process more of the Rust crate?
It might be reasonable for the foreign bindings to know that Rust "paths" to modules which should be processed, and inside those modules find structs "marked up" as being used by the FFI.
In other words, borrowing the example above:
#[uniffi::magic]
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}maybe can be made to work, so long as we are happy to help UniFFI discover where such annotations may exist.
A complication here is that currently UniFFI allows types defined in external crates, but that might still be workable - eg, diplomat has an issue open to support exactly this
When reviewing the draft of this document, @rfk noted that we are already duplicating Rust structs in UDL and in Rust. So instead of having:
// In a UDL file:
dictionary MyFfiType {
i32 a;
bool b;
};
// Then in Rust:
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}
we could have:
// In the Rust implementation, in some other module.
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}
// And to expose it over the FFI:
#[ffi::something]
mod ffi {
#[ffi::magic_external_type_declaration]
pub struct MyFfiType {
pub a: i32,
pub b: bool,
}
impl MyFfiType {
pub fn create() -> MyFfiType { ... }
...
}
}So while we haven't exactly reduced the duplication, we have removed the UDL.
We probably also haven't helped with documentation, because the natural location for
the documentation of MyFfiType is probably at the actual implementation.
While it might not solve all our problems, it is worthy of serious consideration - fewer problems is still a worthwhile goal, and needing a UDL file and parser seems like one worth removing.
We note above that the type universe described by diplomat is somewhat "leaner" than that described by UniFFI, but in general they are very similar. Thus, there might be a future where merging or otherwise creating some interoperability between these type universes might make sense.
It seems likely that this would start to add unwelcome constraints - eg, diplomat would not want its ability to refactor type representations limited by what UniFFI needs.
However, what you could see happening in the future is UniFFI becoming a kind of higher-level wrapper around Diplomat. You can imagine a Diplomat backend for UniFFI that converts a .udl file into a bridge module and then uses the Diplomat toolchain to generate bindings from it, keeping some of the additional affordances/conveniences UniFFI built for its specific use-cases.
As much as some of the UniFFI team dislike the external UDL file, there's no clear path to moving away from it. We could experiment with some of the options above and see if they are both viable and worth the investment for the UniFFI use-cases. That sounds like a long-term goal.
In the short term, the best we can probably do is to enumerate the perceived problems
with the UDL file and try to make them more ergonomic - for example, avoiding repetition of
[Throws=SomeError] would remove alot of noise, and some strategy for generating
documentation might go a long way.