Ceph Cluster setup

[sandertyu]

NFS-client-provisioner has questionable status for future kubernetes support, and ceph is widely used and quite effective, We can look into how to set this up and replace the current NFS client provisioner.

[sandertyu]

https://github.com/openstack/puppet-ceph/blob/master/USECASES.md#i-want-to-try-this-module,-heard-of-ceph,-want-to-see-it-in-action

docs recommended by richard

and here is more complete ceph documentation https://docs.ceph.com/en/latest/

and here https://wiki.openstack.org/wiki/Puppet/ceph-blueprint

[sandertyu]

The Ceph docs recommend using Rook to integrate into kubernetes. Here are the docs. I believe this can be installed with and through kubernetes, so it would run on our existing machines and we don't need additional puppet stuff.

I'm not sure if we'll be running this on new machines, existing ones, or what. There are a couple of computers that are currently not in production and have some issues, we could maybe try and fix those as well if needed.

[TimStewartJ]

After reading a bunch of docs and articles on ceph and kubernetes integration, it seems like using Rook is definitely the way to go if we want to go through with this. It seems easy enough to set up and should handle the communication between k8s and ceph so we don't have to.

One thing I noticed, though, is that ceph derives most of its benefit from being a cluster. Since the majority of our storage capacity is in blackhole and the rest of the existing cluster doesn't have the storage for this use case, it looks like we'll need some major hardware changes.

Some relevant docs I found:

• Using ceph's block devices with kubernetes
• Using ceph for dynamic provisioning assuming ceph cluster is already set up
• Dynamic provisioning with ceph file system
• Someone's random article about using Rook and PVCs