evaluation bug report

[haowang020110]

Hi, there
I found that the evaluation code was wrong.

REIM-NeRF/scripts/evaluate_c3vd.py

Lines 148 to 150 in 7578806

	depth_se = (pred_depthmaps-ref_depthmaps)**2 # Bx1xHW
	depth_mse = torch.mean(depth_se[...,mask], dim=-1)/frame_data['scene_scale'] # Bx1

The frame_data['scene_scale'] is used to recover the gt depth range,
i.e. the real_gt= ref_depthmaps / frame_data['scene_scale']

However due to the square operation in depth_se, the final results should be
depth_mse = torch.mean(depth_se[...,mask], dim=-1)/(frame_data['scene_scale']**2)
the origin code missed a frame_data['scene_scale'] in the denominator so the results reported in the paper was especially small mse=0.013mm^2 (which seemed impossible)
I run a toy experiment to fixed code, which the mse turn to ~1.305mm^2( a more reasonable results)
I am sorry for any inconvience this issue caused and wish you can fixed this small bug.

[dimitrisPs]

Hi Hao,

Thanks for reviewing the code and reporting this issue
I just checked the code and it seems you are right, the MSE computation is not correct and the values should have been computed in the way you suggested.

While the reported results are not in the correct scale, because the same code was used to evaluate all methods, depth accuracy comparisons between different approaches in the paper are still valid.

Finally, if you are researching depth estimation in endoscopy, consider using metrics that can report in distance units (RMSE etc). While MSE is often used to assess distance accuracy, the metric reports values in mm*mm units, which is not meaningful when measuring distance and can be misleading.

[haowang020110]

Hi, there
Yes it seems RMSE is more meaningful
By the way I'd appreciate it if you can process my pull request