Help training out bright reflection...

[pgaston]

I'm trying to evaluate this for usage in a warehouse. It's almost working! Except I continue to get an incorrect reading on bright reflections - see below. Any ideas on how to train this 'away' ?

Note that running this on the given KITTI_HR model didn't have that effect, but introduced others even stranger - e.g., the anomaly in the upper left of the image.

Thoughts? (I just have to significantly up the ante on computing needs to go 10X greater in training...)

Thanks! This is great!
p

Source image:

My trained image: (roughly 30K images as videos)

Same image using KITTI_HR trained model:

[JamieWatson683]

Hi - and thanks for your interest in the paper!

Great to see ManyDepth (kind of) working on another dataset!

Yes, incorrect predictions on reflections like you see are an unfortunate result of training with a self-supervised loss - where the minimum SSIM loss will correspond to an incorrect depth value.

The cost volume in ManyDepth allows the network to make sensible predictions for the upper left since it most likely can get good matches in this region - Monodepth struggles due to ambiguity.

However it is also the cost volume which will hurt predictions for reflections for ManyDepth - the best match in the cost volume will be wrong, since it assumes a static, lambertian world, and the network just trusts the cost volume here. This is very similar to what we observe in the paper with moving objects - and is why we use a teacher network (i.e. MonoDepth2) to constrain the predictions.

Hope this helps!