Using point cloud to seed gaussian splatting

[Dmitry-Filippov-Rival]

I am trying to convert my static render projects to gaussian splats for 6DoF consumption, as they preserve a lot more details compared to other 3D representations such as point clouds. To do this I currently am taking a video of camera orbiting the scene at different elevation and feed the frames into COLMAP to produce a dataset.

A problem I am running into is that when I use COLMAP, the SfM algorithm produces absolute nonsense unless the images are fairly close together (about 3°), which means that I either need to restrict the view directions from which I supply images or run the training on more than 1000 images which takes forever.

From what I understand, gaussian splatting training algorithm only needs COLMAP dataset to seed the training with point cloud of the scene, which I can very easily generate from my render project without needing to use COLMAP for that.

Question is, would it be easy for me to modify the algorithm to take as input a set of images and point cloud rather than a COLMAP dataset?

[jaco001]

--- COLMAP ISN'T YOUR ENEMY --- ;) Work with your dataset better.
Why? Colmap that gives you very few images are the best 'signal' that you have bad dataset and you must learn how to improve your skils.

Few more topics:

• Most important thing from 'Colmap route' are precise cameras.
• You can seed train with random point cloud but without cameras you get medicore output. (no modification is needed)
• 1000 pic taking forever (not a hour, but days) ... you have VRAM issue. Resize pics , select it better and keep all trainning in VRAM (not a swap) and you are golden.
• Colmap are fine with 15-18 deg coverage.
• Still you mad at Colmap? Try 'Kapture' route. GL

[Dmitry-Filippov-Rival]

--- COLMAP ISN'T YOUR ENEMY --- ;) Work with your dataset better. Why? Colmap that gives you very few images are the best 'signal' that you have bad dataset and you must learn how to improve your skils.

Few more topics:

* Most important thing from 'Colmap route' are precise cameras.

* You can seed train with random point cloud but without cameras you get medicore output. (no modification is needed)

* 1000 pic taking forever (not a hour, but days) ... you have VRAM issue. Resize pics , select it better and keep all trainning in VRAM (not a swap) and you are golden.

* Colmap are fine with 15-18 deg coverage.

* Still you mad at Colmap? Try 'Kapture' route. GL

I think you misunderstand my problem. I am trying to create a gaussian splatting scene from a blender project. That means that I already have all the camera position information for my images and can create a point cloud from the project mesh in the same relative coordinate system. In fact, I have generated NeRF synthetic datasets with a similar method before (by using BlenderNeRF extension) and it seems that for gaussian splatting adding a point cloud to that representation would not be that hard.

That being said, is there an easy way to package all this information that I already have (images, camera positions, point cloud) into a dataset readable by this gaussian splatting training implementation?

[Yuxiiin]

Hi, do you have any idea about this problem now? I constructed my own synthetic dataset using meshes and projected images in the same way, but the loss during training process did not change, the rendered scene was all black, and no valid 3d Gaussians seemed to be generated.I suppose this kind of question is worthy of further exploration.

[TurtleZhong]

@Dmitry-Filippov-Rival I think you can constructed a colmap format using the known poses following this tutorial Reconstruct sparse/dense model from known camera poses. also you can combine point clouds from(lidar or other source) with the colmap outputs. them following the COLMAP dataset structure. use the 3dgs to train it.

[Dmitry-Filippov-Rival]

I went with a slightly different method in the end. There are existing Blender plugins to generate NeRF training data as I mentioned before. After further code exploration I have realized that when training from a NeRF dataset the algorithm generates a random point cloud, saves it as points3d.ply and uses that to seed the gaussians. All I had to do is save a .ply representation of my project in the training dataset folder as points3d.ply and make minor adjustments to the file reader (the code on main currently expects color data from the .ply file that you cannot easily bake in blender).

[Dmitry-Filippov-Rival]

For anyone looking to recreate my method, I am using BlenderNeRF to generate training data, as well as render few scenes from likely view directions for testing data (usually around 20-25) . I then use the normal Blender export to get a .ply file of the scene and put it in the dataset as points3d.ply. The resulting dataset folder looks like so

dataset 
--test 
----001.png
----...
--train
----001.png
----...
--points3d.ply
--transform_test.json
--transform_train.json

I also needed to modify scene/dataset_readers.py starting at line 107

def fetchPly(path, points=100000): 
    plydata = PlyData.read(path)
    vertices = np.random.choice(plydata['vertex'], points)
    positions = np.vstack([vertices['x'], vertices['y'], vertices['z']]).T
    try:
        colors = np.vstack([vertices['red'], vertices['green'], vertices['blue']]).T / 255.0
    except:
        colors = np.zeros_like(positions)
    try:
        normals = np.vstack([vertices['nx'], vertices['ny'], vertices['nz']]).T
    except:
        normals = np.zeros_like(positions)
    return BasicPointCloud(points=positions, colors=colors, normals=normals)

I have sampled 100000 points from the mesh vertices because with too many initialization points my resulting gaussian splat folders ended up too large.