Are UP GT-s correct when training with large distortion?

[suvigy]

Hi!

I'm trying to train with simple Division camera model, which can also model wide angle fisheye camera models. I created the training data set with the following params, I just hightlight the relevant parts.

....
camera_model: simple_divisional
parameter_dists:
  vfov:
     ...
       loc: 1.88
       scale: 0.01 
    ....
  k1:
    .....
      loc:  -0.29104758148391235
      scale: 0.002
.....
 shape:
    type: fix
    value:
      - 640
      - 960

In wandb I'm getting the following visualization also including the up gt, pred, confidence:

Do you think the 'Up Gt' is correct? To me, it seems that there is mostly an inner circle within which the up vectors are correct, but outside of this circle, they appear to be incorrect. Or should it be interpreted in a different way?
(As shown in the example, out-of-FoV pixels are not yet masked. The simple_division model sets every pixel as valid during undistortion. I hope the UpGt issue isn't related to this; I plan to address the valid pixels later.)

[veichta]

Hi,
Very interesting! Would be happy to know how it goes.

My best bet at this point is that using such a big FoV together with strong distortions will produce invalid pixels in the distortion function here since you need (1 - 4 * self.k1[..., None, None] * r2) >= 0.

Unfortunately, we never trained on data where this became relevant, so the check_valid is not used in the most of the code.