Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong Result in Depth Map Extraction #3082

Open
ferphy opened this issue Apr 16, 2024 · 2 comments
Open

Wrong Result in Depth Map Extraction #3082

ferphy opened this issue Apr 16, 2024 · 2 comments

Comments

@ferphy
Copy link

ferphy commented Apr 16, 2024

I've been trying to extract the depth map from a NeRF, specifically from the dataset downloaded by running the line: 'ns-download-data nerfstudio --capture-name=poster" and training it with the Nerfacto model. After reading different issues, I've tried 2 different methods:

The first one involved adding within render.py
(https://github.com/nerfstudio-project/nerfstudio/blob/c599d0f3a1580d4276f4f064fb9cb926eb3b1a71/nerfstudio/scripts/render.py#L156C1-L198C51)
Starting from line 157, after the 'if is_depth' check, the call to a function that would store the output_image into a numpy array.
I extracted this from the following issue: (#2147)
This method resulted in the following image, which as you can see, does not provide sufficient information to extract the depth data for each pixel.

depth_map_00011 (1)

The second method involved modifying render.py again, but this time on line 145, that is, before 'render_image = []', adding the following:

Code_3AiTxqiAKa

This second method resulted in new information that does not resemble what I need:

depth_map_00009 (1)

I extracted this from the following issue: (#1388)

I'm not exactly sure what the problem is causing the incorrect extraction of depth map information from my NeRF. I would be grateful for any suggestions to solve this issue.
Thanks.

@kerrj
Copy link
Collaborator

kerrj commented Apr 18, 2024

Does depth look reasonable in the output render of the viewer? also if you render a pointcloud of the scene does that look reasonable? pointcloud rendering uses depth under the hood to deproject points.

It looks like what might be happening is the floor/walls have fuzzy geometry, which can happen in NeRF when there are highly reflective and texture-less surfaces (eg the floor).

@ferphy
Copy link
Author

ferphy commented Apr 23, 2024

Sorry for replying so late. I've been exploring various approaches to achieve a satisfactory outcome. I believe the issue was related to depth accumulation. After removing that parameter, I obtained the following:

depth_map_00056

It's more accurate but it has a really low definition, maybe This could be due to the artifacts surrounding the NeRF which are restricting my range. Are there any solutions to minimize the interaction of artifacts within the depth map range?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants