Image background for `Spatial3DView`

[bertaveira]

Is your feature request related to a problem? Please describe.
Some applications it is not feasible to rewrite the problem in rust/wgsl and fork the rerun framework to render directly in 3D on the browser. For example some neural rendering or AI generation. But one wants to use the 3D viewer to allow the user to pan around or even add native 3D objects like bounding boxes etc...

Describe the solution you'd like
So ideally one would want to be able to add the externally rendered image (for example from a Diffusion model) as background of the Spatial3DView. Also idealy the user could add a callback on view change and decide if and how to change the background image accordingly, although this might be harder since python/rust interoperability. Maybe just probing periodically is fast enough anyway.

Describe alternatives you've considered
I saw the existing fork (Brush) for gaussian splatting as well as the example for gaussian splatting but none of these really tackle this issue it seems. I might be wrong here but it seemed to me that you either use existing primitives like points, or one has to fork rerun and reimplement things in rust natively in rerun. For example if one wants to connect a HuggingFace diffusion model to is this would be massive efforts compared to just being able to tap into the background image of the 3D viewer.

[Wumpf]

Makes sense! Pretty sure we floated a similar idea for allowing to integrate external renderers previously but never wrote anything down.

Ideally, this should be extensible with a depth buffer and maybe even picking information so that things can be integrated more smoothly.

Also idealy the user could add a callback on view change and decide if and how to change the background image accordingly, although this might be harder since python/rust interoperability. Maybe just probing periodically is fast enough anyway.

This is covered over here:

• New API to request userspace-callbacks from interaction events in visualizer #2691

but the described usecase of continuously sending in framebuffer data is important as it motivates all sort of data being sent.

---

Meanwhile, the custom visualizer example I have drafted here gets close and could implement something like this for a Rust application hosting the viewer.