A neural radiance field ( NeRF ) is a simple fully connected network trained to reproduce input views of a single scene using a rendering loss. The network directly maps from spatial location and viewing direction (5D input) to colour and opacity (4D output), acting as the "volume" so we can use volume rendering to render new views differentiably.
Pipeline |
Novel Views |
3D Reconstruction |
- Training graphs
MSE |
PSNR |
-
Device used: TITAN X (Pascal) 250W / 12GiB RAM
-
Time to train: 12h 30m
-
Training Dashboard: https://wandb.ai/stanleyedward/LegoNeRF/runs/h6yb8pnb/overview
Testing Metrics Values Avg MSE 0.0012457877128773586 Avg PSNR 29.200356294523996
-
git clone https://github.com/stanleyedward/nerf_pytorch_lightning.git cd nerf_pytorch_lightning
-
conda env create -f environment.yaml conda activate nerf
-
Download Lego Dataset: https://drive.google.com/drive/folders/1lrDkQanWtTznf48FCaW5lX9ToRdNDF1a
dataset/ ├── dataset_link.md └── lego ├── test ├── train ├── transforms_test.json ├── transforms_train.json ├── transforms_val.json └── val
note:
Setup should be complete
-
In the config.py file
"""------------------------NeRF Config------------------------""" # data IMG_SIZE: int = 400 BATCH_SIZE: int = 3072 ... DEVICES: int = torch.cuda.device_count() MAX_EPOCHS: int = 17
-
Run the train.py script
python train.py
note:
Setup should be complete
-
In the config.py file
"""------------------------NeRF Config------------------------""" ... #eval CKPT_DIR: str = "models/16_epoch_192_bins_400_nerf.ckpt" CHUNK_SIZE: int = 20 # increase chunksize prevent CUDA out of memory errors OUTPUTS_DIR: str = "outputs" #folder you want to save the novel views in
-
Run the eval.py script
python eval.py
@misc{mildenhall2020nerf,
title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
year={2020},
eprint={2003.08934},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
@misc{queianchen_nerf,
author={Quei-An, Chen},
title={Nerf_pl: a pytorch-lightning implementation of NeRF},
url={https://github.com/kwea123/nerf_pl/},
year={2020},
}
@misc{lin2020nerfpytorch,
title={NeRF-pytorch},
author={Yen-Chen, Lin},
publisher = {GitHub},
journal = {GitHub repository},
howpublished={\url{https://github.com/yenchenlin/nerf-pytorch/}},
year={2020}
}