DOC-933 Document new consumer group lag metrics and configs (#1014)

Author: JakeSCahill

modules/get-started/pages/release-notes/redpanda.adoc

@@ -39,6 +39,22 @@ Redpanda now supports normalization of Protobuf schemas in the Schema Registry.
 
 You now can configure Kafka clients to authenticate using xref:manage:security/authentication#enable-sasl.adoc[SASL/PLAIN] with a single account using the same username and password. Unlike SASL/SCRAM, which uses a challenge response with hashed credentials, SASL/PLAIN transmits plaintext passwords. You enable SASL/PLAIN by appending `PLAIN` to the list of SASL mechanisms.
 
+== New metrics
+
+The following metrics are new in this version:
+
+=== Consumer lag gauges
+
+Redpanda can now expose dedicated consumer lag gauges that eliminate the need to calculate lag manually. These metrics provide real-time insights into consumer group performance and help identify issues. The following metrics are available:
+
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_max[`redpanda_kafka_consumer_group_lag_max`]:
+Reports the maximum lag observed among all partitions for a consumer group. This metric helps pinpoint the partition with the greatest delay, indicating potential performance or configuration issues.
+
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_sum[`redpanda_kafka_consumer_group_lag_sum`]:
+Aggregates the lag across all partitions, providing an overall view of data consumption delay for the consumer group.
+
+See xref:manage:monitoring.adoc#consumers[Monitor consumer group lag] for more information.
+
 == New cluster properties
 
 The following cluster properties are new in this version:

modules/manage/partials/monitor-health.adoc

@@ -209,13 +209,132 @@ Leaderless partitions can be caused by unresponsive brokers. When an alert on `r
 
 Redpanda's Raft implementation exchanges periodic status RPCs between a broker and its peers. The xref:reference:public-metrics-reference.adoc#redpanda_node_status_rpcs_timed_out[`redpanda_node_status_rpcs_timed_out`] gauge increases when a status RPC times out for a peer, which indicates that a peer may be unresponsive and may lead to problems with partition replication that Raft manages. Monitor for non-zero values of this gauge, and correlate it with any logged errors or changes in partition replication.
 
-=== Consumers
+[[consumers]]
+=== Consumer group lag
 
-==== Consumer group lag
+Consumer group lag is an important performance indicator that measures the difference between the broker's latest (max) offset and the consumer group's last committed offset. The lag indicates how current the consumed data is relative to real-time production. A high or increasing lag means that consumers are processing messages slower than producers are generating them. A decreasing or stable lag implies that consumers are keeping pace with producers, ensuring real-time or near-real-time data consumption.
 
-When working with Kafka consumer groups, the consumer group lag—the difference between the broker's latest (max) offset and the group's last committed offset—is a performance indicator of how fresh the data being consumed is. While higher lag for archival consumers is expected, high lag for real-time consumers could indicate that the consumers are overloaded and thus may need their topics to be partitioned more, or to spread the load to more consumers.
+By monitoring consumer lag, you can identify performance bottlenecks and make informed decisions about scaling consumers, tuning configurations, and improving processing efficiency.
 
-To monitor consumer group lag, create a query with the xref:reference:public-metrics-reference.adoc#redpanda_kafka_max_offset[`redpanda_kafka_max_offset`] and xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_committed_offset[`redpanda_kafka_consumer_group_committed_offset`] gauges:
+A high maximum lag may indicate that a consumer is experiencing connectivity problems or cannot keep up with the incoming workload.
+
+A high or increasing total lag (lag sum) suggests that the consumer group lacks sufficient resources to process messages at the rate they are produced. In such cases, scaling the number of consumers within the group can help, but only up to the number of partitions available in the topic. If lag persists despite increasing consumers, repartitioning the topic may be necessary to distribute the workload more effectively and improve processing efficiency.
+
+Redpanda provides the following methods for monitoring consumer group lag:
+
+- <<dedicated-gauges, Dedicated gauges>>: Redpanda brokers can internally calculate consumer group lag and expose two dedicated gauges. This method is recommended for environments where your observability platform does not support complex queries required to calculate the lag from offset metrics.
++
+Enabling these gauges may add a small amount of additional processing overhead to the brokers.
+- <<offset-based-calculation, Offset-based calculation>>: You can use your observability platform to calculate consumer group lag from offset metrics. Use this method if your observability platform supports functions, such as `max()`, and you prefer to avoid additional processing overhead on the broker.
+
+==== Dedicated gauges
+
+Redpanda can internally calculate consumer group lag and expose it as two dedicated gauges.
+
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_max[`redpanda_kafka_consumer_group_lag_max`]:
+Reports the maximum lag observed among all partitions for a consumer group. This metric helps pinpoint the partition with the greatest delay, indicating potential performance or configuration issues.
+
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_sum[`redpanda_kafka_consumer_group_lag_sum`]:
+Aggregates the lag across all partitions, providing an overall view of data consumption delay for the consumer group.
+
+To enable these dedicated gauges, you must enable consumer group metrics in your cluster properties. Add the following settings to your Redpanda configuration:
+
+- xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`]: A list of properties to enable for consumer group metrics. You must add the `consumer_lag` property to enable consumer group lag metrics.
+- xref:reference:properties/cluster-properties.adoc#consumer_group_lag_collection_interval_sec[`consumer_group_lag_collection_interval_sec`] (optional): The interval in seconds for collecting consumer group lag metrics. The default is 60 seconds.
++
+Set this value equal to the scrape interval of your metrics collection system. Aligning these intervals ensures synchronized data collection, reducing the likelihood of missing or misaligned lag measurements.
+
+For example:
+
+ifndef::env-kubernetes[]
+[,bash]
+----
+rpk cluster config set enable_consumer_group_metrics '["group", "partition", "consumer_lag"]'
+----
+endif::[]
+
+ifdef::env-kubernetes[]
+[tabs]
+======
+Helm + Operator::
++
+--
+.`redpanda-cluster.yaml`
+[,yaml]
+----
+apiVersion: cluster.redpanda.com/v1alpha2
+kind: Redpanda
+metadata:
+  name: redpanda
+spec:
+  chartRef: {}
+  clusterSpec:
+    config:
+      cluster:
+        enable_consumer_group_metrics:
+          - group
+          - partition
+          - consumer_lag
+----
+
+```bash
+kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
+```
+
+--
+Helm::
++
+--
+[tabs]
+====
+--values::
++
+.`enable-consumer-metrics.yaml`
+[,yaml]
+----
+config:
+  cluster:
+    enable_consumer_group_metrics:
+      - group
+      - partition
+      - consumer_lag
+----
++
+```bash
+helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
+--values enable-consumer-metrics.yaml --reuse-values
+```
+
+--set::
++
+[,bash]
+----
+helm upgrade --install redpanda redpanda/redpanda \
+  --namespace <namespace> \
+  --create-namespace \
+  --set config.cluster.enable_consumer_group_metrics[0]=group \
+  --set config.cluster.enable_consumer_group_metrics[1]=partition \
+  --set config.cluster.enable_consumer_group_metrics[2]=consumer_lag
+----
+
+====
+--
+======
+endif::[]
+
+
+When these properties are enabled, Redpanda computes and exposes the `redpanda_kafka_consumer_group_lag_max` and `redpanda_kafka_consumer_group_lag_sum` gauges to the `/public_metrics` endpoint.
+
+==== Offset-based calculation
+
+If your environment is sensitive to the performance overhead of the <<dedicated-gauges, dedicated gauges>>, use the offset-based calculation method to calculate consumer group lag. This method requires your observability platform to support functions like `max()`.
+
+Redpanda provides two metrics to calculate consumer group lag:
+
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_max_offset[`redpanda_kafka_max_offset`]: The broker's latest offset for a partition.
+- xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_committed_offset[`redpanda_kafka_consumer_group_committed_offset`]: The last committed offset for a consumer group on that partition.
+
+For example, here's a typical query to compute consumer lag:
 
 [,promql]
 ----

modules/reference/pages/properties/cluster-properties.adoc

@@ -414,6 +414,26 @@ This is an internal-only configuration and should be enabled only after consulti
 
 ---
 
+=== consumer_group_lag_collection_interval_sec
+
+How often to run the collection loop when <<enable_consumer_group_metrics,`enable_consumer_group_metrics`>> contains `consumer_lag`.
+
+Reducing the value of `consumer_group_lag_collection_interval_sec` increases the metric collection frequency, which may raise resource utilization. In most environments, this impact is minimal, but it's best practice to monitor broker resource usage in high-scale settings.
+
+*Unit:* seconds
+
+*Requires restart:* No
+
+*Visibility:* `tunable`
+
+*Type:* integer
+
+*Accepted values:* [`-17179869184`, `17179869183`]
+
+*Default:* `60`
+
+---
+
 === controller_backend_housekeeping_interval_ms
 
 Interval between iterations of controller backend housekeeping loop.
@@ -812,6 +832,54 @@ Maximum amount of time the coordinator waits to snapshot after a command appears
 
 ---
 
+=== datalake_scheduler_block_size_bytes
+
+Size, in bytes, of each memory block reserved for record translation, as tracked by the datalake scheduler.
+
+*Unit:* bytes
+
+*Requires restart:* Yes
+
+*Visibility:* `tunable`
+
+*Type:* integer
+
+*Default:* `4_mib`
+
+---
+
+=== datalake_scheduler_max_concurrent_translations
+
+The maximum number of translations that the datalake scheduler will allow to run at a given time. If a translation is requested, but the number of running translations exceeds this value, the request will be put to sleep temporarily, polling until capacity becomes available.
+
+*Requires restart:* Yes
+
+*Visibility:* `tunable`
+
+*Type:* integer
+
+*Default:* `4`
+
+---
+
+=== datalake_scheduler_time_slice_ms
+
+Time, in milliseconds, for a datalake translation as scheduled by the datalake scheduler. After a translation is scheduled, it will run until either the time specified has elapsed or all pending records on its source partition have been translated.
+
+*Unit:* milliseconds
+
+*Requires restart:* Yes
+
+*Visibility:* `tunable`
+
+*Type:* integer
+
+*Accepted values:* [`-17592186044416`, `17592186044415`]
+
+*Default:* `30000`
+
+---
+
 === debug_bundle_auto_removal_seconds
 
 If set, how long debug bundles are kept in the debug bundle storage directory after they are created. If not set, debug bundles are kept indefinitely.
@@ -1061,7 +1129,15 @@ Enables cluster metadata uploads. Required for xref:manage:whole-cluster-restore
 
 === enable_consumer_group_metrics
 
-List of enabled consumer group metrics. Accepted Values: `group`, `partition`, `consumer_lag`.
+List of enabled consumer group metrics. Accepted values include:
+
+- `group`: Enables the xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_consumers[`redpanda_kafka_consumer_group_consumers`] and xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_topics[`redpanda_kafka_consumer_group_topics`] metrics.
+- `partition`: Enables the xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_committed_offset[`redpanda_kafka_consumer_group_committed_offset`] metric.
+- `consumer_lag`: Enables the xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_max[`redpanda_kafka_consumer_group_lag_max`] and xref:reference:public-metrics-reference.adoc#redpanda_kafka_consumer_group_lag_sum[`redpanda_kafka_consumer_group_lag_sum`] metrics
++
+Enabling `consumer_lag` may add a small amount of additional processing overhead to the brokers, especially in environments with a high number of consumer groups or partitions.
++
+Use the xref:reference:properties/cluster-properties.adoc#consumer_group_lag_collection_interval_sec[`consumer_group_lag_collection_interval_sec`] property to control the frequency of consumer lag metric collection.
 
 *Requires restart:* No
 
@@ -1071,6 +1147,9 @@ List of enabled consumer group metrics. Accepted Values: `group`, `partition`, `
 
 *Default:* `["group", "partition"]`
 
+*Related topics*:
+
+- xref:manage:monitoring.adoc#consumers[Monitor consumer group lag]
 ---
 
 === enable_controller_log_rate_limiting
@@ -1712,6 +1791,20 @@ Default value for the `redpanda.iceberg.delete` topic property that determines i
 
 ---
 
+=== iceberg_disable_automatic_snapshot_expiry
+
+Whether to disable automatic Iceberg snapshot expiry. This property may be useful if the Iceberg catalog expects to perform snapshot expiry on its own.
+
+*Requires restart:* No
+
+*Visibility:* `user`
+
+*Type:* boolean
+
+*Default:* `false`
+
+---
+
 === iceberg_disable_snapshot_tagging
 
 Whether to disable tagging of Iceberg snapshots. These tags are used to ensure that the snapshots that Redpanda writes are retained during snapshot removal, which in turn, helps Redpanda ensure exactly-once delivery of records. Disabling tags is therefore not recommended, but may be useful if the Iceberg catalog does not support tags.

modules/reference/pages/public-metrics-reference.adoc

@@ -865,39 +865,82 @@ Histogram of client quota throughput per quota rule and type.
 
 Committed offset for a consumer group, segmented by topic and partition.
 
+To enable this metric, you must include the `partition` option in the xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`] cluster property.
+
 *Type*: gauge
 
 *Labels*:
 
-* `group`
-
-* `topic`
-
-* `partition`
+- `redpanda_group`
+- `redpanda_partition`
+- `redpanda_topic`
+- `shard`
 
 ---
 
 === redpanda_kafka_consumer_group_consumers
 
 Number of active consumers within a consumer group.
 
+To enable this metric, you must include the `group` option in the xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`] cluster property.
+
 *Type*: gauge
 
 *Labels*:
 
-* `group`
+- `redpanda_group`
+- `shard`
+
+---
+
+=== redpanda_kafka_consumer_group_lag_max
+
+Maximum consumer group lag across topic partitions. This metric is useful for identifying the most delayed partition in the consumer group.
+
+To enable this metric, you must include the `consumer_lag` option in the xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`] cluster property.
+
+*Type*: gauge
+
+*Labels*:
+
+- `redpanda_group`
+
+*Related topics*:
+
+- xref:manage:monitoring.adoc#consumer-group-lag[Consumer group lag]
+
+---
+
+=== redpanda_kafka_consumer_group_lag_sum
+
+Sum of consumer group lag for all topic partitions. This metric is useful for tracking the total lag across all partitions.
+
+To enable this metric, you must include the `consumer_lag` option in the xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`] cluster property.
+
+*Type*: gauge
+
+*Labels*:
+
+- `redpanda_group`
+
+*Related topics*:
+
+- xref:manage:monitoring.adoc#consumer-group-lag[Consumer group lag]
 
 ---
 
 === redpanda_kafka_consumer_group_topics
 
 Number of topics being consumed by a consumer group.
 
+To enable this metric, you must include the `group` option in the xref:reference:properties/cluster-properties.adoc#enable_consumer_group_metrics[`enable_consumer_group_metrics`] cluster property.
+
 *Type*: gauge
 
 *Labels*:
 
-* `group`
+- `redpanda_group`
+- `shard`
 
 == REST proxy metrics
 
@@ -1006,7 +1049,7 @@ Build information for Redpanda, including the revision and version details.
 
 ---
 
-=== redpanda_application_fips_mode 
+=== redpanda_application_fips_mode
 
 Indicates whether Redpanda is running in FIPS mode. Possible values: