Clustering Across Multiple Maches

[ctjutting]

I have 2x VMs running Ubuntu Server, Docker, and Gravity. I'm attempting to cluster both instances of Gravity together, but it doesn't appear it will cluster across multiple VMS. Only cluster on the same box. Is this the case? If not, then how does clustering actually work?

There was another question like this earlier this year, and it didn't seem to have any responses from the community or others.

[BeryJu]

Yes, clustering across multiple (virtual) machines is supported (otherwise it would be quite pointless). The clustering is just using etcd's clustering which requires port 2380/TCP to be accessible between different machines.

[ctjutting]

Is there documentation on how to do this? Or is this based on ETCDs docs? I have the cluster environment variables setup in docker compose however the cluster nodes never join.

…

On Tue, Nov 26, 2024 at 4:23 PM Jens L. ***@***.***> wrote: Yes, clustering across multiple (virtual) machines is supported (otherwise it would be quite pointless). The clustering is just using etcd's clustering which requires port 2380/TCP to be accessible between different machines. — Reply to this email directly, view it on GitHub <#1330 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX4HHQ6O2APAH7SIW67Q3Q32CTYENAVCNFSM6AAAAABSRMNARGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZYHA3DSOA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[BeryJu]

The docker compose that is output when you join a node can be used on any other vm that has :2380 available. There are some docs for clustering here: https://gravity.beryju.io/docs/install/cluster/

[ctjutting]

The docs were followed and still unhelpful. They are quite lacking in expected outputs. Neither vm technically joins the cluster as expected.

…

On Tue, Nov 26, 2024 at 6:15 PM Jens L. ***@***.***> wrote: The docker compose that is output when you join a node can be used on any other vm that has :2380 available. There are some docs for clustering here: https://gravity.beryju.io/docs/install/cluster/ — Reply to this email directly, view it on GitHub <#1330 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX4HHQ37QNVBJEKRJY7DRYD2CUFJVAVCNFSM6AAAAABSRMNARGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZYHEZDSMQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tompaah]

I've setup a three-node cluster (three separate VMs) multiple times without problems regarding the clustering.
Maybe I can help you.

Found it easiest to deploy the nodes using ready-made Docker Compose files, particularly regarding the clustering config. That is; create an ADMIN_TOKEN on the first node and the use ETCD_JOIN_CLUSTER on the following nodes to connect to the first node. If you have the Docker Compose files ready to deploy it takes no more than 15 seconds to get a cluster up and running from scratch.

A general reflection is that the UI lacks a few features regarding to the etcd/cluster nodes handling, so if you already joined nodes that need to be removed or messed up some other way, just scrap everything (including persistent storage) and begin all over again. If you already have a lot of data in the database, do an export with the CLI gravity cli export /data/dump.json
as documented at https://gravity.beryju.io/docs/backup/
Note that this dump.json will also contain cluster information, so if you have cluster problems you only want to keep dns/dhcp-data only and need to edit the dump.json. Delete all keys that are not /dns, /discovery or /dhcp prefixed.

I'll provide my Docker Compose files here with some commentary, if they can help you. My three nodes are
gravity01, 192.168.1.2
gravity02, 192.168.1.3
gravity03, 192.168.1.4
The volume mapping is unique for my environment so please change it to whatever you want to use.

Here's the first node. Note the ADMIN_TOKEN, go to https://it-tools.tech/token-generator and generate something for yourself.

services:
  gravity:
    # Important for this to be static and unique
    hostname: gravity01
    image: ghcr.io/beryju/gravity:v0.18.0
    restart: unless-stopped
    network_mode: host
    user: root
    volumes:
      - /volume1/stacks/gravity/data:/data
    environment:
      LOG_LEVEL: info
      INSTANCE_IP: 192.168.1.2
      INSTANCE_IDENTIFIER: gravity01
      DEBUG: false
      ADMIN_PASSWORD: a_predefined_admin_password_of_your_choice
      ADMIN_TOKEN: your_random_admin_token
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  data:
    driver: local

Node two and three are joined with the ETCD_JOIN_CLUSTER variable, using the ADMIN_TOKEN from the first node. They are identical except their IP-addresses and names so I'll only share the second nodes compose file.

services:
  gravity:
    hostname: gravity02
    image: ghcr.io/beryju/gravity:v0.18.0
    restart: unless-stopped
    network_mode: host
    user: root
    environment:
      ETCD_JOIN_CLUSTER: your_random_admin_token,http://192.168.1.2:8008
      BOOTSTRAP_ROLES: dns;dhcp;api;etcd;discovery;backup;monitoring;tsdb;tftp
      INSTANCE_IP: 192.168.1.3
      INSTANCE_IDENTIFIER: gravity02
      DEBUG: false
    volumes:
      - /volume1/stacks/gravity/data:/data
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "3"
volumes:
  data:
    driver: local

Drop a line if this helps.

[ctjutting]

This may be the missing piece I was looking for. I'm only experimenting for right now. The goal is to replace bind and kea. Bind works great, Kea is just awful to contend with. I think this gets me across the finish line

…

On Wed, Nov 27, 2024 at 2:37 AM tompaah ***@***.***> wrote: I've setup a three-node cluster (three separate VMs) multiple times without problems regarding the clustering. Maybe I can help you. Found it easiest to deploy the nodes using ready-made Docker Compose files, particularly regarding the clustering config. That is; create an ADMIN_TOKEN on the first node and the use ETCD_JOIN_CLUSTER on the following nodes to connect to the first node. If you have the Docker Compose files ready to deploy it takes no more than 15 seconds to get a cluster up and running from scratch. A general reflection is that the UI lacks a few features regarding to the etcd/cluster nodes handling, so if you already joined nodes that need to be removed or messed up some other way, just scrap everything (including persistent storage) and begin all over again. If you already have a lot of data in the database, do an export with the CLI gravity cli export /data/dump.json as documented at https://gravity.beryju.io/docs/backup/ Note that this dump.json will also contain cluster information, so if you have cluster problems you only want to keep dns/dhcp-data only and need to edit the dump.json. Delete all keys that are not /dns, /discovery or /dhcp prefixed. I'll provide my Docker Compose files here with some commentary, if they can help you. My three nodes are gravity01, 192.168.1.2 gravity02, 192.168.1.3 gravity03, 192.168.1.4 The volume mapping is unique for my environment so please change it to whatever you want to use. Here's the first node. Note the ADMIN_TOKEN, go to https://it-tools.tech/token-generator and generate something for yourself. services: gravity: # Important for this to be static and unique hostname: gravity01 image: ghcr.io/beryju/gravity:v0.18.0 restart: unless-stopped network_mode: host user: root volumes: - /volume1/stacks/gravity/data:/data environment: LOG_LEVEL: info INSTANCE_IP: 192.168.1.2 INSTANCE_IDENTIFIER: gravity01 DEBUG: false ADMIN_PASSWORD: a_predefined_admin_password_of_your_choice ADMIN_TOKEN: your_random_admin_token logging: driver: json-file options: max-size: "10m" max-file: "3" volumes: data: driver: local Node two and three are joined with the ETCD_JOIN_CLUSTER variable, using the ADMIN_TOKEN from the first node. They are identical except their IP-addresses and names so I'll only share the second nodes compose file. services: gravity: hostname: gravity02 image: ghcr.io/beryju/gravity:v0.18.0 restart: unless-stopped network_mode: host user: root environment: ETCD_JOIN_CLUSTER: your_random_admin_token,http://192.168.1.2:8008 BOOTSTRAP_ROLES: dns;dhcp;api;etcd;discovery;backup;monitoring;tsdb;tftp INSTANCE_IP: 192.168.1.3 INSTANCE_IDENTIFIER: gravity02 DEBUG: false volumes: - /volume1/stacks/gravity/data:/data logging: driver: json-file options: max-size: 10m max-file: "3" volumes: data: driver: local Drop a line if this helps. — Reply to this email directly, view it on GitHub <#1330 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX4HHQYXIVXD2I53L2TX2JL2CWAFLAVCNFSM6AAAAABSRMNARGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZZGIYDSMY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Parrat0]

This is the way indeed. Super simple and refreshingly straight forward. I found that adding nodes after any configuration has been done will not work very well of at all. For cluster setup you need to set them up clean as a cluster: meaning clean/new volumes. Any changes made afterwards will propagate through the cluster nicely.

I did however notice that ADMIN_PASSWORD is only needed on the first node.
INSTANCE_IP should be the primary address of the nodes.
INSTANCE_IDENTIFIER should be unique for each node, I keep them the same as 'hostname'.

Node 1:

services:
  gravity:
    hostname: gravity1
    image: ghcr.io/beryju/gravity:stable
    restart: unless-stopped
    network_mode: host
    volumes:
      - data:/data
    environment:
      LOG_LEVEL: info
      INSTANCE_IP: <ip of first instance; e.g. 10.0.0.1>
      INSTANCE_IDENTIFIER: gravity1
      INSTANCE_LISTEN: 0.0.0.0
      ADMIN_TOKEN: <admin token, either generated from the web like above or like me generated from a previous install>
      ADMIN_PASSWORD: <admin account password>
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  data:

Node 2 & 3:

services:
  gravity:
    hostname: <gravity2 & gravity3>
    image: ghcr.io/beryju/gravity:stable
    restart: unless-stopped
    network_mode: host
    volumes:
      - data:/data
    environment:
      LOG_LEVEL: info
      INSTANCE_IP: <respective instance ips: e.g. 10.0.0.2 & e.g. 10.0.0.3>
      INSTANCE_IDENTIFIER: <respective identifiers; e.g. gravity2 & gravity3>
      INSTANCE_LISTEN: 0.0.0.0
      ETCD_JOIN_CLUSTER: <admin token>,http://10.0.0.1:8008 (use ip of first instance!)
      BOOTSTRAP_ROLES: dns;dhcp;api;etcd;discovery;backup;monitoring;tsdb;tftp
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

volumes:
  data: