DTU Evaluation Metric

[Jiangshuyi0V0]

Thank you for your awesome work. I have a few questions about the DTU-CD evaluation step.

I noticed that the evaluated distance in the code is based on the ground truth point cloud file with reconstructed blocks, excluding the background and floor, while the GT point cloud file includes the background. Could you kindly clarify whether you exclusively use the mean_d2s metric in your paper, or do you take the average of mean_d2s and means2d? If it's the latter, how do you handle the environmental point cloud in the referenced ground truth file?

[monniert]

Hi @Jiangshuyi0V0, sorry for the late reply, I was afk for a few weeks!

We use the standard DTU evaluation which evaluates a given mesh wrt to the GT point cloud files: you can find a python implementation in this codebase, which is a copy from this repo https://github.com/jzhangbs/DTUeval-python. Note that the GT point cloud indeed includes part of the background, but the evaluation script uses object masks (dilated by a few pixels IIRC, more information in the DTU paper: https://roboimagedata2.compute.dtu.dk/data/text/multiViewCVPR2014.pdf) to filter out GT points that do not belong to the object. Therefore, removing the background and floor from our predictions was more aligned with this evaluation process.

We report the average of mean_d2s and mean_s2d