Rust roadmap

[reidlai]

Feature scope

CLI (options, error messages, logging, etc.)

Description

Is there any roadmap for using rust to build tap, target, transform and map?

[edgarrmondragon]

Hey @reidlai!

Nothing as concrete as a roadmap but some folks in the community and myself have done interesting PoCs for Rust-based connectors.

See for example:

• https://github.com/edgarrmondragon/singer-rust
• https://github.com/z3z1ma/singer-rust

In the first example, there's even a toy target published to PyPI thanks to Maturin¹ so it's something you could already add to Meltano! (Taking a personal note to add an example of that to the README).

That said, the main takeaways from those PoCs are:

1. IO is slightly more performant, perhaps in large part because we're deserializing into an struct. The singer-sdk is surely leaving some performance on the table in that regard. See feat: Faster JSON dumps in format_message with msgspec sdk#1784.
2. Rust and Python are miles apart in terms of beginner friendliness, so it's unclear how much the community could participate in creating novel connectors.
3. The point above makes me think that one interesting approach would be to write the most demanded targets and mappers in Rust with CLIs published to PyPI.

---

I'm curious what you think of that, and whether you have expectations of what part Rust would play in Meltano plugins.

Footnotes

1. Build and publish crates with pyo3, rust-cpython, cffi and uniffi bindings as well as rust binaries as python packages with minimal configuration. https://github.com/pyo3/maturin. ↩

[reidlai]

Actually python is not efficient compared with rust. I know that I can use rust to create tap and target using rust...but I can foresee if we can use rust to build custom mapping and transformation, meltano can run very fast without GC