topic's totalSize, storageSize, offloadedStorageSize

[KannarFr]

Hi,

Running pulsar v3.2.0

yo-pulsar-broker-c3-n1 ~ # pulsar-admin topics stats X/Y/Z-partition-0
{
  ...
  "storageSize" : 3 414 640 099 816,
  ...
  "offloadedStorageSize" : 2 877 652 907 299,
  ...
}

According to https://pulsar.apache.org/docs/3.2.x/administration-stats/#topic-stats there is no totalSize in metrics and it should be computed "manually" as totalSize = storageSize (BK) + offloadedStorageSize (offloaded). But I got the following offload policies:

yo-pulsar-broker-c3-n1 ~ # pulsar-admin namespaces get-offload-policies X/Y
{
  "managedLedgerOffloadThresholdInBytes" : 536 870 912 000,
  "managedLedgerOffloadDeletionLagInMillis" : 1800000,

And here storageSize = offloadedStorageSize + managedLedgerOffloadThresholdInBytes, so storageSize looks to be total, right? So the only way to get BK size is storageSize - offloadedStorageSize?

Is there an implementation issue or documentation issue?

[KannarFr]

Also, on BK sizes, is that the replicated sizes?

[dao-jun]

@KannarFr

You are misunderstanding

1. storageSize: total bytes of the topic, includes offloaded and BK
2. offloadedStorageSize: The bytes of data offloaded to tiered-storage.
3. managedLedgerOffloadThresholdInBytes: If the bytes of data in BK exceeds the threshold, offload the data to tiered-storage which is exceeded. Say, If 3Gb data in BK, and managedLedgerOffloadThresholdInBytes is 2Gb, will offload 3Gb-2Gb = 1Gb data to tired-storage. So the storageSize=3Gb, offloadedStorageSize = 1Gb.

Repeat it again, storageSize = BK size + offloadedStorageSize. BK size means that the bytes of data stores in Bookkeeper.z

[KannarFr]

Thanks for your reply.

If you consider Prometheus metrics you are correct, but not the topic stats for storageSize?

[dao-jun]

@KannarFr Prometheus stats and TopicStats are using the same implementation, they should be keep align.

Generally speaking, totalSize is equals to storageSize.

Seems it truly a doc issue, I'll asking other contributors.

[dao-jun]

@KannarFr It confirmed as a doc issue, I've created a PR to fix it. apache/pulsar-site#870