Bytewax Rerun

Modules to make rerun integration easy.

This modules offers a single Sink: RerunSink. This makes it easy to log messages to Rerun from bytewax, properly handling multiple workers. You can instantiate as many sinks as you need, and you have to pass a RerunMessage to them:

op.output("rerun-time-sink", messages_stream, RerunSink("app_id", "recording_id"))

RerunMessage is a helper class that defines an entity that can be logged into Rerun with all the properties needed:

message = RerunMessage(
    entity_path=f"metrics/{name}",
    entity=rr.Scalar(value),
    timeline="metrics",
    time=seconds_since_start,
)

The sink supports all Rerun's operating modes: spawn, connect, save and serve. So you can use the sink to record metrics to a file for each worker, and later use the Rerun viewer to replay all the recordings togheter.

The sink also offers a RerunSink.rerun_log decorator. If you decorate any of your functions with this, Bytewax will log the moment the function was called, and how long it took to run the function into a separate timeline in Rerun. The metrics are divided by worker, so you can see when each one is activated and for how long. You can optionally log the arguments used in each function, so you can see your items flowing through the dataflow.

Setting up the project for development

Install just

We use just as a command runner for actions / recipes related to developing Bytewax. Please follow the installation instructions. There's probably a package for your OS already.

Install pyenv and Python 3.12

I suggest using pyenv to manage python versions. the installation instructions.

You can also use your OS's package manager to get access to different Python versions.

Ensure that you have Python 3.12 installed and available as a "global shim" so that it can be run anywhere. The following will make plain python run your OS-wide interpreter, but will make 3.12 available via python3.12.

$ pyenv global system 3.12

Install uv

We use uv as a virtual environment creator, package installer, and dependency pin-er. There are a few different ways to install it, but I recommend installing it through either brew on macOS or pipx.

Install just

We use just as a command runner for actions / recipes related to developing Bytewax. Please follow the installation instructions. There's probably a package for your OS already.

Development

We have a just recipe that will:

1. Set up a venv in venvs/dev/.

2. Install all dependencies into it in a reproducible way.

Start by adding any dependencies that are needed into pyproject.toml or into requirements/dev.in if they are needed for development.

Next, generate the pinned set of dependencies with

> just venv-compile-all

Create and activate a virtual environment

Once you have compiled your dependencies, run the following:

> just get-started

Activate your development environment and run the development task:

> . venvs/dev/bin/activate
> just develop

License

bytewax-rerun is commercially licensed with publicly available source code. You are welcome to prototype using this module for free, but any use on business data requires a paid license. See https://modules.bytewax.io/ for a license. Please see the full details in LICENSE.