Skip to content

Commit 7b83b26

Browse files
HengyiWangHengyiWang
committed
code release
1 parent 63a1cd3 commit 7b83b26

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+12095
-0
lines changed

‎.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
# MacOS
2+
.vscode
3+
.DS_Store
4+
15
# Byte-compiled / optimized / DLL files
26
__pycache__/
37
*.py[cod]

‎README.md

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM
2+
3+
### [Paper](https://arxiv.org/pdf/2304.14377.pdf) | [Project Page](https://hengyiwang.github.io/projects/CoSLAM) | [Video](https://hengyiwang.github.io/projects/Co-SLAM/videos/presentation.mp4)
4+
5+
> Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM <br />
6+
> [Hengyi Wang](https://hengyiwang.github.io/), [Jingwen Wang](https://jingwenwang95.github.io/), [Lourdes Agapito](http://www0.cs.ucl.ac.uk/staff/L.Agapito/)<br />
7+
> CVPR 2023
8+
9+
<p align="center">
10+
<a href="">
11+
<img src="./media/coslam_teaser.gif" alt="Logo" width="80%">
12+
</a>
13+
</p>
14+
15+
16+
17+
This repository contains the code for the paper Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM, a neural SLAM method that perform real-time camera tracking and dense reconstruction based on a joint encoding.
18+
19+
20+
21+
## Update
22+
23+
- [x] Code for Co-SLAM [2023-5-12]
24+
- [x] Code for offline RGB-D reconstruction [click here](https://github.com/HengyiWang/Hybrid-Surf). [2023-5-12]
25+
- [ ] Code for evaluation strategy, performance analysis [click here](https://github.com/JingwenWang95/neural_slam_eval).
26+
- [ ] Tutorials on creating sequences using iPhone/iPad Pro
27+
28+
## Installation
29+
30+
Please follow the instructions below to install the repo and dependencies.
31+
32+
```bash
33+
git clone https://github.com/HengyiWang/Co-SLAM.git
34+
cd Co-SLAM
35+
```
36+
37+
38+
39+
### Install the environment
40+
41+
```bash
42+
# Create conda environment
43+
conda create -n coslam python=3.7
44+
conda activate coslam
45+
46+
# Install the pytorch first (Please check the cuda version)
47+
pip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu113/torch_stable.html
48+
49+
# Install all the dependencies via pip (Note here pytorch3d and tinycudann requires ~10min to build)
50+
pip install -r requirements.txt
51+
52+
# Build extension (marching cubes from neuralRGBD)
53+
cd external/NumpyMarchingCubes
54+
python setup.py install
55+
56+
```
57+
58+
59+
60+
For tinycudann, if you cannot access network when you use GPUs, you can also try build from source as below:
61+
62+
```bash
63+
# Build tinycudann
64+
git clone --recursive https://github.com/nvlabs/tiny-cuda-nn
65+
66+
# Try this version if you cannot use the latest version of tinycudann
67+
#git reset --hard 91ee479d275d322a65726435040fc20b56b9c991
68+
cd tiny-cuda-nn/bindings/torch
69+
python setup.py install
70+
```
71+
72+
73+
74+
## Dataset
75+
76+
#### Replica
77+
78+
Download the sequences of the Replica Dataset generated by the authors of iMAP into `./data/Replica` folder.
79+
80+
```bash
81+
bash scripts/download_replica.sh # Released by authors of NICE-SLAM
82+
```
83+
84+
85+
86+
#### ScanNet
87+
88+
Please follow the procedure on [ScanNet](http://www.scan-net.org/) website, and extract color & depth frames from the `.sens` file using the [code](https://github.com/ScanNet/ScanNet/blob/master/SensReader/python/reader.py).
89+
90+
91+
92+
#### Synthetic RGB-D dataset
93+
94+
Download the sequences of the synethetic RGB-D dataset generated by the authors of neuralRGBD into `./data/neural_rgbd_data` folder. We exclude the scenes with NaN poses generated by BundleFusion.
95+
96+
```bash
97+
bash scripts/download_rgbd.sh
98+
```
99+
100+
101+
102+
#### TUM RGB-D
103+
104+
Download 3 sequences of TUM RGB-D dataset into `./data/TUM` folder.
105+
106+
```bash
107+
bash scripts/download_tum.sh
108+
```
109+
110+
111+
112+
## Run
113+
114+
You can run Co-SLAM using the code below:
115+
116+
```
117+
python coslam.py --config './configs/{Dataset}/{scene}.yaml
118+
```
119+
120+
121+
122+
You can run Co-SLAM with multi-processing using the code below:
123+
124+
```
125+
python coslam_mp.py --config './configs/{Dataset}/{scene}.yaml
126+
```
127+
128+
129+
130+
## Evaluation
131+
132+
We employ a slightly different evaluation strategy to measure the quality of the reconstruction, you can find out the code [here](https://github.com/JingwenWang95/neural_slam_eval). Note if you want to follow the evaluation protocol of NICE-SLAM, please refer to our supplementary material for detailed parameters setting.
133+
134+
135+
136+
## Acknowledgement
137+
138+
We adapt codes from some awesome repositories, including [NICE-SLAM](https://github.com/cvg/nice-slam), [NeuralRGBD](https://github.com/dazinovic/neural-rgbd-surface-reconstruction), [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn). Thanks for making the code available. We also thank [Zihan Zhu](https://zzh2000.github.io/) of [NICE-SLAM](https://github.com/cvg/nice-slam), [Edgar Sucar](https://edgarsucar.github.io/) of [iMAP](https://edgarsucar.github.io/iMAP/) for their prompt responses to our inquiries regarding the details of their methods.
139+
140+
The research presented here has been supported by a sponsored research award from Cisco Research and the UCL Centre for Doctoral Training in Foundational AI under UKRI grant number EP/S021566/1. This project made use of time on Tier 2 HPC facility JADE2, funded by EPSRC (EP/T022205/1).
141+
142+
143+
144+
## Citation
145+
146+
If you find our code or paper useful for your research, please consider citing:
147+
148+
```
149+
@inproceedings{wang2023coslam,
150+
title={Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM},
151+
author={Wang, Hengyi and Wang, Jingwen and Agapito, Lourdes},
152+
booktitle={CVPR},
153+
year={2023}
154+
}
155+
```
156+
157+

‎config.py

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
import yaml
2+
3+
4+
def load_config(path, default_path=None):
5+
"""
6+
Loads config file.
7+
Args:
8+
path (str): path to config file.
9+
default_path (str, optional): whether to use default path. Defaults to None.
10+
Returns:
11+
cfg (dict): config dict.
12+
"""
13+
# load configuration from file itself
14+
with open(path, 'r') as f:
15+
cfg_special = yaml.full_load(f)
16+
17+
# check if we should inherit from a config
18+
inherit_from = cfg_special.get('inherit_from')
19+
20+
# if yes, load this config first as default
21+
# if no, use the default_path
22+
if inherit_from is not None:
23+
cfg = load_config(inherit_from, default_path)
24+
elif default_path is not None:
25+
with open(default_path, 'r') as f:
26+
cfg = yaml.full_load(f)
27+
else:
28+
cfg = dict()
29+
30+
# include main configuration
31+
update_recursive(cfg, cfg_special)
32+
33+
return cfg
34+
35+
36+
def update_recursive(dict1, dict2):
37+
"""
38+
Update two config dictionaries recursively.
39+
Args:
40+
dict1 (dict): first dictionary to be updated.
41+
dict2 (dict): second dictionary which entries should be used.
42+
"""
43+
for k, v in dict2.items():
44+
if k not in dict1:
45+
dict1[k] = dict()
46+
if isinstance(v, dict):
47+
update_recursive(dict1[k], v)
48+
else:
49+
dict1[k] = v

‎configs/Azure/apartment.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
inherit_from: configs/Azure/azure.yaml
2+
mapping:
3+
bound: [[-1.5,3.5],[-4.5,3.6],[-6.5,8.]]
4+
marching_cubes_bound: [[-1.5,3.5],[-4.5,3.6],[-6.5,8.]]
5+
6+
data:
7+
datadir: data/Apartment
8+
trainskip: 1
9+
output: output/Apartment
10+
exp_name: demo

‎configs/Azure/azure.yaml

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
dataset: 'azure'
2+
3+
data:
4+
downsample: 1
5+
sc_factor: 1
6+
translation: 0
7+
num_workers: 4
8+
9+
mapping:
10+
sample: 2048
11+
first_mesh: True
12+
iters: 10
13+
lr_embed: 0.01
14+
lr_decoder: 0.01
15+
lr_rot: 0.001
16+
lr_trans: 0.001
17+
keyframe_every: 5
18+
map_every: 5
19+
n_pixels: 0.05
20+
first_iters: 500
21+
optim_cur: True
22+
min_pixels_cur: 100
23+
map_accum_step: 1
24+
pose_accum_step: 5
25+
map_wait_step: 0
26+
filter_depth: False
27+
28+
tracking:
29+
iter: 10
30+
sample: 1024
31+
pc_samples: 40960
32+
lr_rot: 0.001
33+
lr_trans: 0.001
34+
ignore_edge_W: 20
35+
ignore_edge_H: 20
36+
iter_point: 0
37+
wait_iters: 100
38+
const_speed: True
39+
best: True
40+
41+
grid:
42+
enc: 'HashGrid'
43+
tcnn_encoding: True
44+
hash_size: 16
45+
voxel_color: 0.08
46+
voxel_sdf: 0.02
47+
oneGrid: True
48+
49+
pos:
50+
enc: 'OneBlob'
51+
n_bins: 16
52+
53+
decoder:
54+
geo_feat_dim: 15
55+
hidden_dim: 32
56+
num_layers: 2
57+
num_layers_color: 2
58+
hidden_dim_color: 32
59+
tcnn_network: False
60+
61+
62+
cam:
63+
H: 720
64+
W: 1280
65+
fx: 607.4694213867188
66+
fy: 607.4534912109375
67+
cx: 636.9967041015625
68+
cy: 369.2689514160156
69+
png_depth_scale: 1000. #for depth image in png format
70+
crop_edge: 0
71+
near: 0
72+
far: 5
73+
depth_trunc: 100.
74+
75+
76+
training:
77+
rgb_weight: 5.0
78+
depth_weight: 0.1
79+
sdf_weight: 1000
80+
fs_weight: 10
81+
eikonal_weight: 0
82+
smooth_weight: 0.001 #0.001
83+
smooth_pts: 64
84+
smooth_vox: 0.1
85+
smooth_margin: 0.05
86+
#n_samples: 256
87+
n_samples_d: 64
88+
range_d: 0.2
89+
n_range_d: 21
90+
n_importance: 0
91+
perturb: 1
92+
white_bkgd: False
93+
trunc: 0.1
94+
rot_rep: 'axis_angle'
95+
rgb_missing: 0.0
96+
97+
mesh:
98+
resolution: 512
99+
vis: 500
100+
voxel_eval: 0.05
101+
voxel_final: 0.03
102+
visualisation: False
103+

‎configs/Replica/office0.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
inherit_from: configs/Replica/replica.yaml
2+
mapping:
3+
bound: [[-3,3],[-4,2.5],[-2,2.5]]
4+
marching_cubes_bound: [[-2.2,2.6],[-3.4,2.1],[-1.4,2.0]]
5+
6+
7+
data:
8+
datadir: data/Replica/office0
9+
trainskip: 1
10+
output: output/Replica/office0
11+
exp_name: demo

‎configs/Replica/office1.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
inherit_from: configs/Replica/replica.yaml
2+
mapping:
3+
bound: [[-2,3.2],[-1.7,2.7],[-1.2,2.0]]
4+
marching_cubes_bound: [[-1.9,3.1],[-1.6,2.6],[-1.1,1.8]]
5+
6+
7+
8+
data:
9+
datadir: data/Replica/office1
10+
trainskip: 1
11+
output: output/Replica/office1
12+
exp_name: demo

‎configs/Replica/office2.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
inherit_from: configs/Replica/replica.yaml
2+
mapping:
3+
bound: [[-3.6,3.2],[-3.0,5.5],[-1.4,1.7]]
4+
marching_cubes_bound: [[-3.5,3.1],[-2.9,5.4],[-1.3,1.6]]
5+
6+
7+
data:
8+
datadir: data/Replica/office2
9+
trainskip: 1
10+
output: output/Replica/office2
11+
exp_name: demo

‎configs/Replica/office3.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
inherit_from: configs/Replica/replica.yaml
2+
mapping:
3+
bound: [[-5.3,3.7],[-6.1,3.4],[-1.4,2.0]]
4+
marching_cubes_bound: [[-5.2,3.6],[-6.0,3.3],[-1.3,1.9]]
5+
6+
7+
8+
data:
9+
datadir: data/Replica/office3
10+
trainskip: 1
11+
output: output/Replica/office3
12+
exp_name: demo

‎configs/Replica/office4.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
inherit_from: configs/Replica/replica.yaml
2+
mapping:
3+
bound: [[-1.4,5.5],[-2.5,4.4],[-1.4,1.8]]
4+
marching_cubes_bound: [[-1.3,5.4],[-2.4,4.3],[-1.3,1.7]]
5+
6+
7+
data:
8+
datadir: data/Replica/office4
9+
trainskip: 1
10+
output: output/Replica/office4
11+
exp_name: demo

0 commit comments

Comments
 (0)