Skip to content

Latest commit

 

History

History
86 lines (67 loc) · 9.06 KB

reference-metrics.md

File metadata and controls

86 lines (67 loc) · 9.06 KB
Raw
id title sidebar_label
reference-metrics
KoP Metrics
KoP Metrics
<style type="text/css"> table{ font-size: 80%; } </style>

KoP exposes the following metrics in Prometheus format. You can monitor your clusters with those metrics.

The following types of metrics are available:

  • Counter: a cumulative metric that represents a single monotonically increasing counter. The value increases by default. You can reset the value to zero or restart your cluster.
  • Gauge: a metric that represents a single numerical value that can arbitrarily go up and down.
  • Histogram: a histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets.
  • Summary: similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

KoP metrics

The KoP metrics are exposed under "/metrics" at port 8080 along with Pulsar metrics. You can use a different port by configuring the stats_server_port system property.

Channel metrics

Name Type Description
kop_server_ALIVE_CHANNEL_COUNT Gauge The number of alive request channel
kop_server_ACTIVE_CHANNEL_COUNT Gauge The number of active request channel

Request metrics

Name Type Description
kop_server_REQUEST_QUEUE_SIZE Gauge The number of quest in kop request processing queue of total request channel.
kop_server_REQUEST_QUEUED_LATENCY Summary The requests queued latency calculated in milliseconds.
Available labels: request (ApiVersions, Metadata, Produce, FindCoordinator, ListOffsets, OffsetFetch, OffsetCommit, Fetch, JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, ListGroups, DeleteGroups, SaslHandshake, SaslAuthenticate, CreateTopics, InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, TxnOffsetCommit, EndTxn, WriteTxnMarkers, DescribeConfigs, DeleteTopics).
kop_server_REQUEST_PARSE_LATENCY Summary The requests parse latency from byteBuf to MemoryRecords calculated in milliseconds.
kop_server_REQUEST_LATENCY Summary The requests processing total latency for all Kafka Apis.
Available labels: request (ApiVersions, Metadata, Produce, FindCoordinator, ListOffsets, OffsetFetch, OffsetCommit, Fetch, JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, ListGroups, DeleteGroups, SaslHandshake, SaslAuthenticate, CreateTopics, InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, TxnOffsetCommit, EndTxn, WriteTxnMarkers, DescribeConfigs, DeleteTopics).

Response metrics

Name Type Description
kop_server_RESPONSE_BLOCKED_TIMES Counter The response blocked times due to waiting for process complete
kop_server_RESPONSE_BLOCKED_LATENCY Summary The response blocked latency calculated in milliseconds

Producer metrics

Name Type Description
kop_server_PENDING_TOPIC_LATENCY Summary The latency for when a pending topic future finishes
kop_server_PRODUCE_ENCODE Summary The memory record encode latency
kop_server_MESSAGE_PUBLISH Summary The message publish latency to Pulsar ManagedLedger
kop_server_MESSAGE_QUEUED_LATENCY Summary The message queued latency in KoP message publish queue
kop_server_BYTES_IN Counter The producer bytes in stats.
Available labels: topic, partition.
  • topic: the topic name to produce.
  • partition: the partition id for the topic to produce
kop_server_MESSAGE_IN Counter The producer message in stats.
Available labels: topic, partition.
  • topic: the topic name to produce.
  • partition: the partition id for the topic to produce
kop_server_BATCH_COUNT_PER_MEMORYRECORDS Gauge The number of batches in each memory records
kop_server_PRODUCE_MESSAGE_CONVERSIONS Counter The producer message conversions in stats.
Available labels: topic, partition.
  • topic: the topic name to produce.
  • partition: the partition id for the topic to produce
kop_server_PRODUCE_MESSAGE_CONVERSIONS_TIME_NANOS Summary The producer message convert latency in nanoseconds.
Available labels: topic, partition.
  • topic: the topic name to produce.
  • partition: the partition id for the topic to produce

Consumer metrics

Name Type Description
kop_server_PREPARE_METADATA Summary The prepare metadata latency in milliseconds before starting fetch from Pulsar ManagedLedger
kop_server_TOTAL_MESSAGE_READ Summary The total message read latency in milliseconds in this fetch request
kop_server_MESSAGE_READ Summary The message read latency in milliseconds for one cursor read entry request
kop_server_FETCH_DECODE Summary The message decode latency in milliseconds
kop_server_BYTES_OUT Counter The consumer bytes out stats.
Available labels: topic, partition, group.
  • topic: the topic name to consume.
  • partition: the partition id for the topic to consume
  • group: the group id for consumer to consumer message from topic-partition
kop_server_MESSAGE_OUT Counter The consumer message out stats.
Available labels: topic, partition, group.
  • topic: the topic name to consume.
  • partition: the partition id for the topic to consume
  • group: the group id for consumer to consumer message from topic-partition
kop_server_ENTRIES_OUT Counter The consumer entries out stats.
Available labels: topic, partition, group.
  • topic: the topic name to consume.
  • partition: the partition id for the topic to consume
  • group: the group id for consumer to consumer message from topic-partition
kop_server_CONSUME_MESSAGE_CONVERSIONS Counter The consumer message conversions in stats.
Available labels: topic, partition.
  • topic: the topic name to consume.
  • partition: the partition id for the topic to consume
kop_server_CONSUME_MESSAGE_CONVERSIONS_TIME_NANOS Summary The consumer message convert latency in nanoseconds.
Available labels: topic, partition.
  • topic: the topic name to consume.
  • partition: the partition id for the topic to consume
kop_server_WAITING_FETCHES_TRIGGERED Counter Number of fetches that have been delayed due to not enough data, and that have been unblocked because some message has been produced

Kop event metrics

Name Type Description
kop_server_KOP_EVENT_QUEUE_SIZE Gauge The total number of events in KoP event processing queue.
kop_server_KOP_EVENT_QUEUED_LATENCY Summary The events queued latency calculated in milliseconds.
Available labels: event (DeleteTopicsEvent, BrokersChangeEvent, ShutdownEventThread).
kop_server_KOP_EVENT_LATENCY Summary The events processing total latency for all KoP event types.
Available labels: event (DeleteTopicsEvent, BrokersChangeEvent, ShutdownEventThread).