From 42e48d648ef32144c2d3c6f98cfbcea63a001560 Mon Sep 17 00:00:00 2001 From: nolouch Date: Thu, 15 May 2025 17:36:55 +0800 Subject: [PATCH 1/6] Add VM switch guide for TiUP cluster Signed-off-by: nolouch --- maintain-tidb-using-tiup.md | 169 ++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index fad48716104f4..2a960c7db930a 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -290,3 +290,172 @@ The destroy operation stops the services and clears the data directory and deplo ```bash tiup cluster destroy ${cluster-name} ``` + +## Switch from Prometheus to VictoriaMetrics + +In large clusters, Prometheus may face efficiency challenges, especially when there are many instances. Since TiUP v1.16.3, TiUP supports switching the metrics server from Prometheus to VictoriaMetrics (VM) to provide better scalability, higher performance, and lower resource consumption. + +### Set up VictoriaMetrics for a new deployment + +By default, TiUP uses Prometheus as the metrics server. To use VictoriaMetrics instead of Prometheus in a new deployment, configure the topology file as follows: + +```yaml +# Monitoring server configuration +monitoring_servers: + # IP address of the monitoring server + - host: ip_address + ... + prom_remote_write_to_vm: true + enable_prom_agent_mode: true + +# Grafana server configuration +grafana_servers: + # IP address of the Grafana server + - host: ip_address + ... + use_vm_as_datasource: true +``` + +### Migrate an existing deployment to VictoriaMetrics + +The migration process can be performed without affecting running instances. Existing metrics will remain in Prometheus, while new metrics will be written to VictoriaMetrics. + +#### Enable VictoriaMetrics remote write + +1. Edit the cluster configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster edit-config ${cluster-name} + ``` + +2. Under `monitoring_servers`, set `prom_remote_write_to_vm` to `true`: + + ```yaml + monitoring_servers: + - host: ip_address + ... + prom_remote_write_to_vm: true + ``` + +3. Reload the updated configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster reload ${cluster-name} -R prometheus + ``` + +#### Switch the default data source to VictoriaMetrics + +1. Edit the cluster configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster edit-config ${cluster-name} + ``` + +2. Under `grafana_servers`, set `use_vm_as_datasource` to `true`: + + ```yaml + grafana_servers: + - host: ip_address + ... + use_vm_as_datasource: true + ``` + +3. Reload the updated configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster reload ${cluster-name} -R grafana + ``` + +#### (Optional) View historical metrics generated before the switch + +If you need to view historical metrics generated before the switch, you can switch the Grafana data source as follows: + +1. Edit the cluster configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster edit-config ${cluster-name} + ``` + +2. Comment out `use_vm_as_datasource` under `grafana_servers`: + + ```yaml + grafana_servers: + - host: ip_address + ... + # use_vm_as_datasource: true + ``` + +3. Reload the updated configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster reload ${cluster-name} -R grafana + ``` + +4. To switch back to VictoriaMetrics, repeat the steps in "Switch the default data source to VictoriaMetrics". + +### Clean up old metrics and services + +After confirming that old metrics have expired, you can remove redundant services and files as follows. This will not affect the running cluster. + +#### Set Prometheus to agent mode + +1. Edit the cluster configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster edit-config ${cluster-name} + ``` + +2. Under `monitoring_servers`, set `enable_prom_agent_mode` to `true`, and ensure `prom_remote_write_to_vm` and `use_vm_as_datasource` are also set correctly: + + ```yaml + monitoring_servers: + - host: ip_address + ... + prom_remote_write_to_vm: true + enable_prom_agent_mode: true + grafana_servers: + - host: ip_address + ... + use_vm_as_datasource: true + ``` + +3. Reload the updated configuration: + + {{< copyable "shell-regular" >}} + + ```bash + tiup cluster reload ${cluster-name} -R prometheus + ``` + +#### Remove expired data directories + +1. Find the `data_dir` of the monitoring server in the configuration file: + + ```yaml + monitoring_servers: + - host: ip_address + ... + data_dir: "/tidb-data/prometheus-8249" + ``` + +2. Remove the data directory: + + {{< copyable "shell-regular" >}} + + ```bash + rm -rf /tidb-data/prometheus-8249 + ``` From 10725e6001edf84841af4c267856a172539aba17 Mon Sep 17 00:00:00 2001 From: ShuNing Date: Tue, 20 May 2025 16:03:31 +0800 Subject: [PATCH 2/6] Update maintain-tidb-using-tiup.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- maintain-tidb-using-tiup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index 2a960c7db930a..e3b5338f7037e 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -419,7 +419,7 @@ After confirming that old metrics have expired, you can remove redundant service tiup cluster edit-config ${cluster-name} ``` -2. Under `monitoring_servers`, set `enable_prom_agent_mode` to `true`, and ensure `prom_remote_write_to_vm` and `use_vm_as_datasource` are also set correctly: +2. Under `monitoring_servers`, set `enable_prom_agent_mode` to `true`, and ensure you also set `prom_remote_write_to_vm` and `use_vm_as_datasource` correctly: ```yaml monitoring_servers: From 612ffc950ebd4ed6fbda0497f60fe839582f09c5 Mon Sep 17 00:00:00 2001 From: ShuNing Date: Tue, 20 May 2025 16:03:39 +0800 Subject: [PATCH 3/6] Update maintain-tidb-using-tiup.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- maintain-tidb-using-tiup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index e3b5338f7037e..63b36cc545b54 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -318,7 +318,7 @@ grafana_servers: ### Migrate an existing deployment to VictoriaMetrics -The migration process can be performed without affecting running instances. Existing metrics will remain in Prometheus, while new metrics will be written to VictoriaMetrics. +You can perform the migration process without affecting running instances. Existing metrics will remain in Prometheus, and TiUP will write new metrics to VictoriaMetrics. #### Enable VictoriaMetrics remote write From e27b20cc415d4723b50a4eb23e093f8eb20acb32 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 28 May 2025 17:32:59 +0800 Subject: [PATCH 4/6] Update format --- maintain-tidb-using-tiup.md | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index 63b36cc545b54..a70bf4b4fcc3e 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -324,8 +324,6 @@ You can perform the migration process without affecting running instances. Exist 1. Edit the cluster configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster edit-config ${cluster-name} ``` @@ -341,8 +339,6 @@ You can perform the migration process without affecting running instances. Exist 3. Reload the updated configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster reload ${cluster-name} -R prometheus ``` @@ -351,8 +347,6 @@ You can perform the migration process without affecting running instances. Exist 1. Edit the cluster configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster edit-config ${cluster-name} ``` @@ -368,8 +362,6 @@ You can perform the migration process without affecting running instances. Exist 3. Reload the updated configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster reload ${cluster-name} -R grafana ``` @@ -380,8 +372,6 @@ If you need to view historical metrics generated before the switch, you can swit 1. Edit the cluster configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster edit-config ${cluster-name} ``` @@ -397,8 +387,6 @@ If you need to view historical metrics generated before the switch, you can swit 3. Reload the updated configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster reload ${cluster-name} -R grafana ``` @@ -413,8 +401,6 @@ After confirming that old metrics have expired, you can remove redundant service 1. Edit the cluster configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster edit-config ${cluster-name} ``` @@ -435,8 +421,6 @@ After confirming that old metrics have expired, you can remove redundant service 3. Reload the updated configuration: - {{< copyable "shell-regular" >}} - ```bash tiup cluster reload ${cluster-name} -R prometheus ``` @@ -454,8 +438,6 @@ After confirming that old metrics have expired, you can remove redundant service 2. Remove the data directory: - {{< copyable "shell-regular" >}} - ```bash rm -rf /tidb-data/prometheus-8249 ``` From 6d154f51fc2daa652c4d49ad0a2b9d8acb89598f Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 28 May 2025 17:35:11 +0800 Subject: [PATCH 5/6] Simplify the beginning intro --- maintain-tidb-using-tiup.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index a70bf4b4fcc3e..fd8126c025f7a 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -6,14 +6,7 @@ aliases: ['/docs/dev/maintain-tidb-using-tiup/','/docs/dev/how-to/maintain/tiup- # TiUP Common Operations -This document describes the following common operations when you operate and maintain a TiDB cluster using TiUP. - -- View the cluster list -- Start the cluster -- View the cluster status -- Modify the configuration -- Stop the cluster -- Destroy the cluster +This document describes the common operations when you operate and maintain a TiDB cluster using TiUP. ## View the cluster list From 4702353d890cf1607721fc4987d3bce8ee21085d Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 28 May 2025 17:36:17 +0800 Subject: [PATCH 6/6] Update format --- maintain-tidb-using-tiup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/maintain-tidb-using-tiup.md b/maintain-tidb-using-tiup.md index fd8126c025f7a..a47f256ea8bea 100644 --- a/maintain-tidb-using-tiup.md +++ b/maintain-tidb-using-tiup.md @@ -359,7 +359,7 @@ You can perform the migration process without affecting running instances. Exist tiup cluster reload ${cluster-name} -R grafana ``` -#### (Optional) View historical metrics generated before the switch +#### View historical metrics generated before the switch (optional) If you need to view historical metrics generated before the switch, you can switch the Grafana data source as follows: