Release v5.0.0

Logging Core Module Release v5.0.0

Welcome to the latest release of the logging module of SIGHUP Distribution maintained by team SIGHUP by ReeVo.

This update is a major version that adds support for the Kubernetes version 1.32 and drops support for 1.28. It also updates all components of module to the latest versions, including new features, security patches and bug fixes.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.19.1	2.17.1
opensearch-dashboards	v2.19.1	2.17.1
logging-operator	v5.2.0	4.10.0
loki-distributed	v3.4.2	2.9.10
minio-ha	RELEASE.2025-02-28T09-55-16Z	RELEASE.2024-10-13T13-34-11Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.32
• Updated MinIO to version RELEASE.2025-02-28T09-55-16Z
• Updated Logging Operator to version 5.2.0
• Added a ServiceMonitor resource for Logging Operator
• Updated Loki to version 3.4.2
• Updated OpenSearch and OpenSearch Dashboards images to 2.19.1
• Added a ServiceMonitor for OpenSearch Dashboards

Breaking Changes 💔

Starting with the v5.0.0 of the Logging Core Module Loki version has been bumped to 3.4.2. Please refer to loki documentation
for the complete release notes.

Update Guide 🦮

Before upgrading Loki from 2.9.10 to 3.4.2

When upgrading Loki the Ingester StatefulSet needs to be scaled to at least 2 replicas before executing the upgrade to avoid losing logs that have not been flushed to a remote storage yet. Notice that the Ingester has an HPA defined and could already be scaled.

Once the StatefulSet has been scaled a patch needs to be applied to add the -ingester.flush-on-shutdown=true and the -log.level=debug flags on the Ingester. This will allow the Ingester to flush logs that still need to be pushed to the long term storage.

You can achieve this by using the following shell script:

ingester_replicas=$(kubectl get statefulsets.apps -n logging loki-distributed-ingester -o jsonpath={.status.currentReplicas})

if [ "${ingester_replicas}" -lt "2" ]; then
  kubectl scale sts -n logging loki-distributed-ingester --replicas=2
  kubectl wait -n logging statefulset/loki-distributed-ingester --for=jsonpath='{.status.availableReplicas}'=2 --timeout=5m
fi

kubectl patch statefulset loki-distributed-ingester -n logging --type='json' -p="[{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/args\",\"value\":[\"-config.file=/etc/loki/config/config.yaml\",\"-ingester.flush-on-shutdown=true\",\"-log.level=debug\",\"-target=ingester\"]}]"

kubectl rollout status -n logging statefulset/loki-distributed-ingester

Once the StatefulSet is stable and the patch has been applied the upgrade can be executed.

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is recommended to use it with the SIGHUP Distribution.

To upgrade the module run:

kustomize build | kubectl apply -f - --server-side

Preview v5.0.0-rc.0

Logging Core Module Release v5.0.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a major version that adds support for the Kubernetes version 1.32.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.19.1	2.17.1
opensearch-dashboards	v2.19.1	2.17.1
logging-operator	v5.2.0	4.10.0
loki-distributed	v3.4.2	2.9.10
minio-ha	RELEASE.2025-02-28T09-55-16Z	RELEASE.2024-10-13T13-34-11Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.32
• Updated MinIO to version RELEASE.2025-02-28T09-55-16Z
• Updated Logging Operator to version 5.2.0
• Added a ServiceMonitor resource for Logging Operator
• Updated Loki to version 3.4.2
• Updated OpenSearch and OpenSearch Dashboards images to 2.19.1
• Added a ServiceMonitor for OpenSearch Dashboards

Breaking Changes 💔

Starting with the v4.1.0 of the Logging Core Module Loki version has been bumped to 3.4.2. Please refer to loki documentation
for the complete release notes.

Update Guide 🦮

Before upgrading Loki from 2.9.10 to 3.4.2

When upgrading Loki the Ingester StatefulSet needs to be scaled to at least 2 replicas before executing the upgrade to avoid losing logs that have not been flushed to a remote storage yet. Notice that the ingester has an HPA defined and could already be scaled.

# check the number of current replicas:
$ kubectl get statefulsets.apps -n logging loki-distributed-ingester -o jsonpath={.status.currentReplicas}
# scale up if necessary:
kubectl scale sts -n logging loki-distributed-ingester --replicas=2

Once the StatefulSet has been scaled a patch needs to be applied to add the -ingester.flush-on-shutdown=true and the -log.level=debug flags on the Ingester. This will allow the ingester to flush logs that still need to be pushed to the long term storage. The patch can be applied with the following command:

kubectl patch statefulset loki-distributed-ingester -n logging --type='json' -p="[{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/args\",\"value\":[\"-config.file=/etc/loki/config/config.yaml\",\"-ingester.flush-on-shutdown=true\",\"-log.level=debug\",\"-target=ingester\"]}]"

Once the StatefulSet is stable and the patch has been applied the upgrade can be executed.

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is reccommended to use fury distribution

To upgrade the module run:

kustomize build | kubectl apply -f -

Release v4.0.0

Logging Core Module Release 4.0.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a major version that adds support for the Kubernetes version 1.30 and 1.31. It also introduces a breaking change with the new time series database for Loki TSDB along with the new v13 schema.
These changes will improve efficiency, speed and scalability of Loki clusters.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

• New schema and store for Loki: The Loki configuration has been extended to add the new TSDB store and v13 schema. There are two new elements in the configuration (note: this is a breaking change):

  ...
  schema_config:
    ...
    - from: "2024-12-01"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v13
      store: tsdb
  ...
  storage_config:
    ...
    tsdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      resync_interval: 5s
      shared_store: s3

  The two pieces of new configuration introduced are:

  • The new schema config with TSDB and schema v13
  • The tsdb_shipper conf that configures the storage for TSDB

  The from in schema_config should be a string in ISO 8601 date format and it represents the day starting from which Loki will record logs with the new store and schema.

  ℹ️ Note: Loki will assume the start of the day on the UTC midnight of the specified day.

Breaking Changes 💔

• Loki store and schema change: A new store and schema has been introduced in order to improve efficiency, speed and scalability of Loki clusters.

Update Guide 🦮

⚠ Before upgrading read about the changes in Loki schema and store

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is reccommended to use fury distribution

To upgrade the module run:

kustomize build | kubectl apply -f -

Preview v4.0.0-rc.3

Logging Core Module Release 4.0.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a major version that adds support for the Kubernetes version 1.30 and 1.31. It also introduces a breaking change with the new time series database for Loki TSDB along with the new v13 schema.
These changes will improve efficiency, speed and scalability of Loki clusters.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

• New schema and store for Loki: The Loki configuration has been extended to add the new TSDB store and v13 schema. There are two new elements in the configuration (note: this is a breaking change):

  ...
  schema_config:
    ...
    - from: "2024-12-01"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v13
      store: tsdb
  ...
  storage_config:
    ...
    tsdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      resync_interval: 5s
      shared_store: s3

  The two pieces of new configuration introduced are:

  • The new schema config with TSDB and schema v13
  • The tsdb_shipper conf that configures the storage for TSDB

  The from in schema_config should be a string in ISO 8601 date format and it represents the day starting from which Loki will record logs with the new store and schema.

  ℹ️ Note: Loki will assume the start of the day on the UTC midnight of the specified day.

Breaking Changes 💔

• Loki store and schema change: A new store and schema has been introduced in order to improve efficiency, speed and scalability of Loki clusters.

Update Guide 🦮

⚠ Before upgrading read about the changes in Loki schema and store

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is reccommended to use fury distribution

To upgrade the module run:

kustomize build | kubectl apply -f -

Preview v4.0.0-rc.1

Logging Core Module Release 4.0.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a major version that adds support for the Kubernetes version 1.30 and 1.31. It also introduces a breaking change with the new time series database for Loki TSDB along with the new v13 schema.
These changes will improve efficiency, speed and scalability of Loki clusters.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

• New schema and store for Loki: The Loki configuration has been extended to add the new TSDB store and v13 schema. There are two new elements in the configuration (note: this is a breaking change):

  ...
  schema_config:
    ...
    - from: "2024-12-01"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v13
      store: tsdb
  ...
  storage_config:
    ...
    tsdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      resync_interval: 5s
      shared_store: s3

  The two pieces of new configuration introduced are:

  • The new schema config with TSDB and schema v13
  • The tsdb_shipper conf that configures the storage for TSDB

  The from in schema_config should be a string in ISO 8601 date format and it represents the day starting from which Loki will record logs with the new store and schema.

  ℹ️ Note: Loki will assume the start of the day on the UTC midnight of the specified day.

Breaking Changes 💔

• Loki store and schema change: A new store and schema has been introduced in order to improve efficiency, speed and scalability of Loki clusters.

Update Guide 🦮

⚠ Before upgrading read about the changes in Loki schema and store

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is reccommended to use fury distribution

To upgrade the module run:

kustomize build | kubectl apply -f -

Preview v4.0.0-rc.0

Logging Core Module Release 4.0.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a major version that adds support for the Kubernetes version 1.30 and 1.31. It also introduces a breaking change with the new time series database for Loki TSDB along with the new v13 schema.
These changes will improve efficiency, speed and scalability of Loki clusters.

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

• New schema and store for Loki: The Loki configuration has been extended to add the new TSDB store and v13 schema. There are two new elements in the configuration (note: this is a breaking change):

  ...
  schema_config:
    ...
    - from: "2024-12-01"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v13
      store: tsdb
  ...
  storage_config:
    ...
    tsdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      resync_interval: 5s
      shared_store: s3

  The two pieces of new configuration introduced are:

  • The new schema config with TSDB and schema v13
  • The tsdb_shipper conf that configures the storage for TSDB

  The from in schema_config should be a string in ISO 8601 date format and it represents the day starting from which Loki will record logs with the new store and schema.

  ℹ️ Note: Loki will assume the start of the day on the UTC midnight of the specified day.

Breaking Changes 💔

• Loki store and schema change: A new store and schema has been introduced in order to improve efficiency, speed and scalability of Loki clusters.

Update Guide 🦮

⚠ Before upgrading read about the changes in Loki schema and store

Upgrade using the distribution

To upgrade the module using the distribution please refer to the official documentation

Manual Upgrade

ℹ️ Note: Manually upgrading the module is deprecated. It is reccommended to use fury distribution

To upgrade the module run:

kustomize build | kubectl apply -f -

Preview v3.5.0-rc.1

Logging Core Module Release 3.5.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a minor patch with a little fix on ingress fluentd outputs

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

Update Guide 🦮

Process

To upgrade the module run:

kustomize build | kubectl apply -f -

Preview v3.5.0-rc.0

Logging Core Module Release 3.5.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a minor patch with a little fix on ingress fluentd outputs

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.17.1	2.12.0
opensearch-dashboards	v2.17.1	2.12.0
logging-operator	v4.10.0	4.5.6
loki-distributed	v2.9.10	2.9.2
minio-ha	RELEASE.2024-10-13T13-34-11Z	RELEASE.2023-01-12T02-06-16Z

Bug Fixes and Changes 🐛

• Added support for Kubernetes version 1.30 and 1.31

Update Guide 🦮

Process

To upgrade the module run:

kustomize build | kubectl apply -f -

Release v3.4.1

Logging Core Module Release 3.4.1

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update is a minor patch with a little fix on ingress fluentd outputs

Component Images 🚢

Component	Supported Version	Previous Version
opensearch	v2.12.0	No update
opensearch-dashboards	v2.12.0	No update
logging-operator	v4.5.6	No update
loki-distributed	v2.9.2	No update
minio-ha	vRELEASE.2023-01-12T02-06-16Z	No update

Bug Fixes and Changes 🐛

• fix: keep k8s labels for ingress-flow @ralgozino

Update Guide 🦮

Process

To upgrade the module run:

kustomize build | kubectl apply -f -

Release v3.4.0

Logging Core Module Release 3.4.0

Welcome to the latest release of the logging module of Kubernetes Fury Distribution maintained by team SIGHUP.

This update includes new versions of various components, removes Cerebro, and adds support for Kubernetes 1.28 and 1.29.

Component Images 🚢

Component	Supported Version	Previous Version
cerebro	removed	0.9.4
opensearch	v2.12.0	2.11.0
opensearch-dashboards	v2.12.0	2.11.0
logging-operator	v4.5.6	4.4.1
loki-distributed	v2.9.2	No update
minio-ha	vRELEASE.2023-01-12T02-06-16Z	No update

Bug Fixes and Changes 🐛

• Added support for Kubernetes versions 1.28 and 1.29.
• Removed Cerebro
• Add positiondb hostPath /var/log/infra-fluentbit-pos to persist fluentbit log forward history

Update Guide 🦮

Process

To upgrade the module, follow the next steps:

1. If you had Cerebro installed, you may want to remove it from the cluster using the copy in your vendor folder:

kustomize build vendor/katalog/logging/cerebro | kubectl delete -f -

2. Download the new version of the module
3. Update the rest of the packages:

kustomize build | kubectl apply -f -

Remove Cerebro after upgrading

If you don't have a local copy of Cerebro on your vendor folder, run the following commands to remove it:

kubectl delete deployment -n logging cerebro
kubectl delete service -n logging cerebro

Identify the name for Cerebro's configmap and delete it:

# get all the configmaps that the name starts with cerebro-config
kubectl get configmap -n logging -ojson | jq -r '.items[] | select(.metadata.name | startswith("cerebro-config")) | .metadata.name'
# validate that the output it valid, then remove all of them
kubectl get configmap -n logging -ojson | jq -r '.items[] | select(.metadata.name | startswith("cerebro-config")) | .metadata.name' | xargs kubectl delete configmap -n logging