diff --git a/docs/cloud/cluster.md b/docs/cloud/cluster.md index 73e2dc1..f0db531 100644 --- a/docs/cloud/cluster.md +++ b/docs/cloud/cluster.md @@ -21,14 +21,14 @@ The distribution/sharding is `fingerprint` based to consist session to the same 1. Create a compose yaml file with two clickhouse servers: ```yaml -version: '2.1' +version: '3.8' networks: common: driver: bridge services: clickhouse: - image: clickhouse/clickhouse-server:22.6.4.35 + image: altinity/clickhouse-server:23.8.11.29.altinitystable container_name: clickhouse.min.deploy hostname: clickhouse.min.deploy domainname: clickhouse.min.deploy @@ -40,7 +40,7 @@ services: - ./config.xml:/etc/clickhouse-server/config.xml - ./_clickhouse-data:/var/lib/clickhouse clickhouse2: - image: clickhouse/clickhouse-server:22.6.4.35 + image: altinity/clickhouse-server:23.8.11.29.altinitystable container_name: clickhouse2.min.deploy hostname: clickhouse2.min.deploy domainname: clickhouse2.min.deploy @@ -55,7 +55,7 @@ services: 2. Export the default config.xml file from clickhouse server docker image: ``` -docker run -it clickhouse/clickhouse-server:22.6.4.35 cat /etc/clickhouse-server/config.xml >config.xml +docker run -it clickhouse/clickhouse-server cat /etc/clickhouse-server/config.xml >config.xml ``` 3. `vi config.xml` or use your preferred text editor diff --git a/docs/cloud/collector.md b/docs/cloud/collector.md index bd65975..f69b583 100644 --- a/docs/cloud/collector.md +++ b/docs/cloud/collector.md @@ -44,7 +44,7 @@ docker pull qxip/qryn-cloud-collector Define a docker compose file to configure each element. ```yml -version: '3.1' +version: '3.8' services: qryn-collector: diff --git a/docs/cloud/config.md b/docs/cloud/config.md index 6a348b9..743238b 100644 --- a/docs/cloud/config.md +++ b/docs/cloud/config.md @@ -16,10 +16,116 @@ Stack configuration is 100% based on [ENV](env.md) parameters passed to the proc +#### ** qryn-ctrl ** + + +See an example, where you use ENV vars as defined on the URL above: +``` + qryn-ctrl: + image: ${version_qryn_ctrl} + container_name: qryn-ctrl + depends_on: + clickhouse-server: + condition: service_healthy + environment: + - QRYN_DATABASE_DATA_0_NAME=${qryn_db_name:-qryn} + - QRYN_DATABASE_DATA_0_HOST=${clickhost:-clickhouse-server} + - QRYN_DATABASE_DATA_0_PORT=${clickhouse_port:-9000} + - QRYN_DATABASE_DATA_0_USER=${clickuser:-default} + - QRYN_DATABASE_DATA_0_PASS=${clickpass:-clickPass} + - QRYN_DATABASE_DATA_0_DEBUG=${qryn_debug:-true} + - QRYN_DATABASE_DATA_0_TTL_DAYS=${qryn_db_ttl:-7} + - QRYN_DATABASE_DATA_0_SECURE=${qryn_db_secure:-false} + - QRYN_DATABASE_DATA_0_ASYNC_INSERT=${qryn_db_async_insert:-false} + command: + - sh + - -c + - ./qryn-ctrl -initialize_db +``` + #### ** qryn-go ** +See an example, where you use ENV vars as defined on the URL above: +``` + qryn-go: + image: ${version_qryn_reader} + container_name: qryn-go + restart: unless-stopped + volumes: + - qryn_reader_data:/var/lib/.prof + - qryn_reader_data:/var/lib/.local + expose: + - ${qryn_go_port} + ports: + - 3200:${qryn_go_port} + environment: + - QRYNCLOUD_LICENSE=${qryn_license} + - QRYN_SYSTEM_SETTINGS_LICENSE_AUTO_SHUTDOWN=${qryn_license_auto_shutdown:-true} + - QRYN_DATABASE_DATA_0_NAME=${qryn_db_name:-qryn} + - QRYN_DATABASE_DATA_0_HOST=${clickhost:-clickhouse-server} + - QRYN_DATABASE_DATA_0_PORT=${clickhouse_port:-9000} + - QRYN_DATABASE_DATA_0_DEBUG=${qryn_debug:-true} + - QRYN_DATABASE_DATA_0_USER=${clickuser:-default} + - QRYN_DATABASE_DATA_0_PASS=${clickpass:-clickPass} + - QRYN_HTTP_SETTINGS_PORT=${qryn_go_port:-3200} + - QRYN_HTTP_SETTINGS_DEBUG=${qryn_http_debug:-true} + - QRYN_LOG_SETTINGS_STDOUT=${qryn_log_stdout:-true} + - QRYN_LOG_SETTINGS_LEVEL=${qryn_log_lvl:-debug} + depends_on: + clickhouse-server: + condition: service_healthy + qryn-ctrl: + condition: service_completed_successfully + logging: + driver: "local" + options: + max-size: "10m" # Rotate logs when they reach 10MB + max-file: "5" # Keep up to 5 rotated log files +``` + #### ** qryn-writer ** +See an example, where you use ENV vars as defined on the URL above: +``` + qryn-writer: + image: ${version_qryn_writer} + container_name: qryn-writer + restart: unless-stopped + volumes: + - qryn_writer_data:/var/lib/.prof + - qryn_writer_data:/var/lib/.local + expose: + - ${qryn_writer_port} + ports: + - 3100:${qryn_writer_port} + environment: + - QRYNCLOUD_LICENSE=${qryn_license} + - QRYN_SYSTEM_SETTINGS_LICENSE_AUTO_SHUTDOWN=${qryn_license_auto_shutdown:-true} + - QRYN_DATABASE_DATA_0_NAME=${qryn_db_name:-qryn} + - QRYN_DATABASE_DATA_0_HOST=${clickhost:-clickhouse-server} + - QRYN_DATABASE_DATA_0_PORT=${clickhouse_port:-9000} + - QRYN_DATABASE_DATA_0_DEBUG=${qryn_debug:-true} + - QRYN_DATABASE_DATA_0_USER=${clickuser:-default} + - QRYN_DATABASE_DATA_0_PASS=${clickpass:-clickPass} + - QRYN_HTTP_SETTINGS_PORT=${qryn_writer_port:-3100} + - QRYN_LOG_SETTINGS_STDOUT=${qryn_log_stdout:-true} + - QRYN_LOG_SETTINGS_LEVEL=${qryn_log_lvl:-debug} + command: + - sh + - -c + - ./cloki-writer + depends_on: + clickhouse-server: + condition: service_healthy + qryn-ctrl: + condition: service_completed_successfully + logging: + driver: "local" + options: + max-size: "10m" # Rotate logs when they reach 10MB + max-file: "5" # Keep up to 5 rotated log files +``` + diff --git a/docs/cloud/env.md b/docs/cloud/env.md index 19bbc6f..2c5ec8e 100644 --- a/docs/cloud/env.md +++ b/docs/cloud/env.md @@ -20,6 +20,20 @@ The following `ENV` parameters can be used to configure and control **qryn:cloud +#### ** qryn-ctrl ** + +| ENV |Default |Usage | +|------------------------|--- |--- | +| QRYN_DATABASE_DATA_0_NAME | qryn | qryn DB name | +| QRYN_DATABASE_DATA_0_HOST | "http://localhost" | Clickhouse Server address | +| QRYN_DATABASE_DATA_0_PORT | 9000 | Clickhouse Binary Port | +| QRYN_DATABASE_DATA_0_USER | default | Clickhouse Username | +| QRYN_DATABASE_DATA_0_PASS | qryn | Clickhouse Password | +| QRYN_DATABASE_DATA_0_DEBUG | false | Set to true to see debug messages about database in log | +| QRYN_DATABASE_DATA_0_TTL_DAYS | 0 | Set the data rotation policy - aka for how long to keep data before pruning it. Most be overrided. | +| QRYN_DATABASE_DATA_0_SECURE | false | Set to true when Clickhouse is hosted on HTTPS (optional) | +| QRYN_DATABASE_DATA_0_ASYNC_INSERT | false | Set to true if you want to ingest data into the DB asyncronously to data being received | + #### ** qryn-go ** diff --git a/docs/cloud/installation.md b/docs/cloud/installation.md index 0cd39e1..256c193 100644 --- a/docs/cloud/installation.md +++ b/docs/cloud/installation.md @@ -8,7 +8,7 @@ !> qryn:cloud requires activation & pull tokens! Please [contact us](mailto:info@qxip.net) to obtain a license. -?> ClickHouse Server LTS version >=22 and <=22.9 is suggested. _22.10+ not yet supported._ +?> ClickHouse Server LTS version >=23 is suggested. ## 📦 Installation @@ -52,7 +52,7 @@ Define a docker compose file to configure each element. ?> Adjust the clickhouse server and authentication details to match your setup ```yml -version: '3.1' +version: '3.8' services: qryn-ctrl: @@ -162,6 +162,10 @@ Use `Kubernetes` and `helm` to get started using either a local or cloud ClickHo ?> You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. +Follow this [guide](https://github.com/metrico/qryn-cloud-helm) to get started with K8s helm charts. + +**Any other steps below show you how to run Qryn in K8s without Helm.** + ##### Pull the qryn service containers ```bash docker pull qxip/qryn-writer-cloud @@ -342,7 +346,10 @@ dpkg -i qryn-writer-cloud_vx.x.x_amd64.deb dpkg -i qryn-go-cloud_vx.x.x_amd64.deb ``` ##### Configure +Configure per [here](./config.md) ```bash +vim /etc/qryn-writer/qryn-writer.json +vim /etc/qryn-go/qryn-go.json ``` ##### Run ```bash @@ -372,7 +379,7 @@ dpkg -i qryn-writer-cloud_vx.x.x_x86_64.rpm dpkg -i qryn-go-cloud_vx.x.x_x86_64.rpm ``` ##### Configure -Configure per [here](/cloud/config.md) +Configure per [here](./config.md) ```bash vim /etc/qryn-writer/qryn-writer.json vim /etc/qryn-go/qryn-go.json @@ -414,7 +421,7 @@ The default rotation mechanism is configured via amount of days for each node of QRYN_DATABASE_DATA_0_TTL_DAYS = 10 ``` -?> The default days value is "7" +?> The default days value is "0". It **must** be configured during the setup. ### Space based rotation