Added a learnable pure background color for splatfacto

[KevinXu02]

Added a learnable pure background color for splatfacto. Useful when the bg is pure color.

[KevinXu02]

Wondering if i should seperate get_outputs_for_camera from get_outputs to git rid of lots of if self.training. And viser currently overwrite bg color in set_background() thus requiring resetting the training/rendering bg every time.

[jb-ye]

Do we need to(self.device) here? I would assume self.learnable_bg will be converted to self.device automatically as part of model.to(device) in init call of VanillaPipeline.

[KevinXu02]

testing on this

[jb-ye]

The PR looks almost good to me (except one minor comment), but it will need to wait until the next gsplat version update before merging this PR (maybe in two weeks or so?).

@KevinXu02 Do you have some experiments that demonstrate the auto background method learns the proper background of the scene (e.g. blue sky)?

[KevinXu02]

The PR looks almost good to me (except one minor comment), but it will need to wait until the next gsplat version update before merging this PR (maybe in two weeks or so?).

@KevinXu02 Do you have some experiments that demonstrate the auto background learns the proper background of the scene (e.g. blue sky)?

this seems trying to simutanously learning the color of the ground and the sky, thus more likely result in a color in the middle. so this might only makes learning the objects floating in a pure color background easier.
I'm currently working on predict 2d bg and that helps a lot in cleaning up floaters, but my current approach is still a bit slow and i'm still optimizing it.

[KevinXu02]

Here's the bicycle scene predict 2d bg instead

[jb-ye]

My general thoughts on this: allowing background to be learnable should be used with some regularizations (e.g. minimizing L1 norm of accumulation map etc.) Otherwise this still is uncontrollable. Let's say you captured a tree from ground, the image should have both leaf, branch, and sky. Would it be nice to absorb sky into the background parameter, otherwise the model would have blue blobs between tree branches, which is super weird.

[KevinXu02]

My general thoughts on this: allowing background to be learnable should be used with some regularizations (e.g. minimizing L1 norm of accumulation map etc.) Otherwise this still is uncontrollable. Let's say you captured a tree from ground, the image should have both leaf, branch, and sky. Would it be nice to absorb sky into the background parameter, otherwise the model would have blue blobs between tree branches, which is super weird.

I’m currently using mlp for this without much encodings and works more likely a tiny nerf, so it mainly tends to represent low frequency stuffs, and always works ok if captured enough amount of sky in multiple views, as the gradient is scaled by the final alpha, so it will not try to learn the foreground. But it would also tends to overkill large gaussians on the ground(which I think abs gs may help?

[jb-ye]

@KevinXu02 The PR will be merged when we produced a new version of gsplat library that support background parameter grad.

[KevinXu02]

@KevinXu02 The PR will be merged when we produced a new version of gsplat library that support background parameter grad.

Got it. Thank you!