About evaluation on P3M500

[felix-ky]

Hi, thnaks for releasing the great repo and paper.
I evaluate matteformer and vitmatte (both using pretrained weight) on P3M500, but both model's MSE error (times 1e3, computed on unknown areas) are 10 times of your result in Table 4, so are you scaling 1e2 for mse error in unknown areas?

[JingfengYao]

Hi, thnaks for releasing the great repo and paper. I evaluate matteformer and vitmatte (both using pretrained weight) on P3M500, but both model's MSE error (times 1e3, computed on unknown areas) are 10 times of your result in Table 4, so are you scaling 1e2 for mse error in unknown areas?

Hi, thanks for your question.

The evaluation code given by ViTMatte evaluates MSE on only trimap unknown region. However, since the methods in Table 4 are not all trimap guided, it is unfair to evaluate in that way. So our results are evaluated on the whole image.

As ViTMatte for instance, for same the inference result, MSE would be 13.17 on unknown regions and be 1.24 on the whole image.