Glomap catastrophic outlier poses

[pablovela5620]

I've been doing a bunch of testing with glomap v.s. colmap and have found a few problems.
linking a few relevant issues

nerfstudio-project/nerfstudio#3409 (comment)
#128

In particular for novel view synthesis using splatfacto from nerfstudio. I'm using a dataset that consists of using the aliked feature detector + lightglue matcher using sequential pairs (from a self-collected video). I've attached a video of whats happening. I'm basically logging the 3d points, camera poses and projected 2d points onto the image.

glomap-failure-featureless.webm

What you can see is that with glomap there's a point where the triangulated points are extremely sparse and it leads to the camera exploding outward to some very far location. Even though the majority of other cameras are okay, I run the exact same feature+matcher but used colmap mapper and didn't have this problem. I'm guessing this is part of the robustness issue between incremental v.s. global triangulation. These outlier poses can really hurt NVS as you can see in this tensorboard log (where I'm getting 11 PSNR v.s. 20 PSNR on glomap | colmap respectively)

I've included the database + images here for debugging purposes. I also included the debug visualization. If using the visualization link (under rerun.io/viewer) make sure to wait a little bit for it to load and click on the frames timeline like the attached screenshot

Glomap reconstruction (catastrophic failure for nvs)

https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/tree/main/250/glomap-aliked-n16-aliked%2Blightglue

https://rerun.io/viewer?url=https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/resolve/main/250-glomap-aliked-n16-aliked%2Blightglue.rrd

Colmap reconstruction (works just fine!)

https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/tree/main/250/colmap-aliked-n16-aliked%2Blightglue

https://rerun.io/viewer?url=https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/resolve/main/250-colmap-aliked-n16-aliked%2Blightglue.rrd

[ichsan2895]

From my experince, glomap is not good when I use sequential matcher. Exhausive matcher seems the best despite it was very slow. If you use hloc, try to use pairs_from_exhausive instead of pairs_from_retrieval / pairs_from_sequential.

[ahojnnes]

After running retrieval or sequential matcher, it can help to run a round of transitive matching to form more triplets.

[pablovela5620]

From my experince, glomap is not good when I use sequential matcher. Exhausive matcher seems the best despite it was very slow. If you use hloc, try to use pairs_from_exhausive instead of pairs_from_retrieval / pairs_from_sequential.

Yea, I was trying to avoid exhaustive since it makes things so much slower. I'll try again with glomap to see if it consistently makes a difference, but colmap worked just fine without exhaustive.

After running retrieval or sequential matcher, it can help to run a round of transitive matching to form more triplets.

Sorry, I'm still a bit new to all this. Can you explain what you mean by transitive matching? Right now the pair selector I'm using does basically

1. Sliding window of size 10 where I create a pair for an image and its next sequential 10 neighbors
2. Loop closure like mechanism after sequential pairs, where I use netvlad every 5 images to find its 5 nearest.

this is the actual pair selector code https://github.com/pablovela5620/hloc-glomap/blob/main/hloc_glomap/pairs_from_sequential.py

and and video that shows what that looks like

Screen.Recording.2024-12-04.at.8.26.36.AM.mp4

[pablovela5620]

Colmap gui also does a good job of showing how one ends up with the extreme outliers with glomap

glomap-outlier-cameras.mp4

[lpanaf]

Hi @pablovela5620, thanks for sharing the data, do you mind providing the full set of images? Now there are only 100 images

[pablovela5620]

Hi @pablovela5620, thanks for sharing the data, do you mind providing the full set of images? Now there are only 100 images

No problem! Images are in that same hf dataset https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/tree/main/250/images

[lpanaf]

Hi @pablovela5620, has the change been pushed to the repo? I still only see 100 images there

[pablovela5620]

@lpanaf apologies! Should be fixed now, not sure why not all the images uploaded
https://huggingface.co/datasets/pablovela5620/glomap-colmap-failure-bedroom/tree/main/250/images

[lpanaf]

Hi @pablovela5620, thank you very much for sharing the data. I have tried to reconstruct the scene with COLMAP features. It seems to successfully registered 253 images

For the provided database, indeed, I also observed some outliers in the result. It is possible to remove them with glomap mapper_resume --skip_pruning 0 --skip_global_positioning 1 --skip_bundle_adjustment 1 --output_path PRUNED_PATH --input_path INPUT_DIR. Outliers should be mostly pruned in this step. Then, to get the camera pose for all images, you can run colmap image_registrator --input_path PRUNED_PATH --database_path DATABASE_PATH --output_path OUTPUT_PATH. The resulting reconstruction would look like this

I tried to compare this result with the colmap reconstruction, the should be close enough

Rotation errors (degrees)
Min:    0.0159179
Max:    2.04118
Mean:   0.308369
Median: 0.241912
P90:    0.509843
P99:    1.9402

Projection center errors
Min:    0.00175209
Max:    0.383553
Mean:   0.035863
Median: 0.0234677
P90:    0.0647614
P99:    0.233893

We will try to add this to the official pipeline as well.

[pablovela5620]

Hi @pablovela5620, thank you very much for sharing the data. I have tried to reconstruct the scene with COLMAP features. It seems to successfully registered 253 images

For the provided database, indeed, I also observed some outliers in the result. It is possible to remove them with glomap mapper_resume --skip_pruning 0 --skip_global_positioning 1 --skip_bundle_adjustment 1 --output_path PRUNED_PATH --input_path INPUT_DIR. Outliers should be mostly pruned in this step. Then, to get the camera pose for all images, you can run colmap image_registrator --input_path PRUNED_PATH --database_path DATABASE_PATH --output_path OUTPUT_PATH. The resulting reconstruction would look like this

I tried to compare this result with the colmap reconstruction, the should be close enough

Rotation errors (degrees)
Min:    0.0159179
Max:    2.04118
Mean:   0.308369
Median: 0.241912
P90:    0.509843
P99:    1.9402

Projection center errors
Min:    0.00175209
Max:    0.383553
Mean:   0.035863
Median: 0.0234677
P90:    0.0647614
P99:    0.233893

We will try to add this to the official pipeline as well.

Great, I will do some testing myself! Thank you for helping me debug. Could you give me some context on how/why this works? Why does skipping bundle adjustment/global positioning help? Is this pruning step done after doing the first initial round with glomap?

[lpanaf]

@pablovela5620 yes, you are right that I run these two steps on top of your provided GLOMAP reconstruction. Starting from scratch, you can directly perform pruning by setting --skip_pruning=0, and you can follow this by colmap image_registrator --input_path PRUNED_PATH --database_path DATABASE_PATH --output_path OUTPUT_PATH

[pablovela5620]

Got it, thank you again! Will let you know how it goes on my end

[ichsan2895]

@lpanaf

glomap mapper_resume --skip_pruning 0 --skip_global_positioning 1 --skip_bundle_adjustment 1 --output_path PRUNED_PATH --input_path INPUT_DIR

Can you give brief explanation why --skip_pruning 0, --skip_global_positioning 1, etc has impact with outlier?

What happen if we set that to default option and apply it to other good reconstruction?

[lpanaf]

Hi @ichsan2895, in the pruning step, GLOMAP tries to separate images into clusters by co-visibile points. So if the reconstruction is well connected, this flag should not have any impact on the final result. As for skip_global_positioning, I set this flag to 1 in my previous reply only because I want to check solely for the pruning step, so we do not need to perform the global positioning step again. If you are running from scratch, please do not skip this step.

[pablovela5620]

Hi @ichsan2895, in the pruning step, GLOMAP tries to separate images into clusters by co-visibile points. So if the reconstruction is well connected, this flag should not have any impact on the final result. As for skip_global_positioning, I set this flag to 1 in my previous reply only because I want to check solely for the pruning step, so we do not need to perform the global positioning step again. If you are running from scratch, please do not skip this step.

So it seems like what is fixing things is the colmap image registrator?

[lpanaf]

@pablovela5620 more precisely, after the pruning step in GLOMAP, some of the images are missing from the reconstruction (outliers images), so to get camera pose for all cameras, it needs to run the COLMAP image_registrator

[pablovela5620]

Great, if you'd like to close this issue I think its basically solved (unless you want to keep it up for now for the more official process you'd mentioned). I can reopen it should I find more problems!

[jtressle]

@lpanaf, even though this is closed, I wanted to share my experience. I've been experimenting with alternative pose initializations feeding into glomap.

For a couple of datasets, I found that outlier images occurred due to incorrect initial poses, causing issues when computing the initial tracks. I was able to correct the outlier images by improving the initial poses. Note during my experimentation, I kept the relative pose estimation and fundamental matrix computations the same, and just toggled initial poses.

So maybe for this dataset, the relative pose estimations are a little off, especially considering some of the images in the dataset could just have been of textureless walls. Could this be an issue where maybe the bundle adjuster isn't robust enough to bad initial poses?

Thanks

[lpanaf]

Hi @jtressle, thanks for sharing your experience. "For a couple of datasets, I found that outlier images occurred due to incorrect initial poses, causing issues when computing the initial tracks. " I agree with you on this part, and I think this is the major problem for the failure cases. But regarding your question "... maybe the bundle adjuster isn't robust enough to bad initial poses?", I think increasing the robustness of BA cannot solve the problem. As you said, when the initial estimation is largely off, the tracks will also be biased to the wrong pose. In this case, there would be very little or no "correct signal" but a more consistent "wrong signal" for the BA to figure out the correct position. So even if the BA is very robust, it would still be almost impossible to get to the correct pose

[jtressle]

@lpanaf thanks for the reply and explanations. I did get better outlier rejection using a tukey loss, but I have a feeling the Huber loss will be more robust overall. I'll open a new issue if I have more concrete observations.

Thanks!