I've been working for a few months on a big rework of the telemetry/metrics system inside of Zeek. It's been through a few iterations, and I'm at a point where I have some unanswerable questions so I'm bringing them up in a ticket. The current design looks like this:

1. Everything is based on the prometheus-cpp library underneath. I originally tried to use opentelemetry but ran into a bunch of shortcomings with their library and documentation and decided to move away from it.
2. All of the existing metrics code moves out of broker/CAF into a framework in Zeek itself. That includes all of the script-land code and options.
3. The built-in aggregation of metrics to the manager node no longer exists. Each node now presents an endpoint for requesting metrics data for Prometheus. There is now a service discovery endpoint on the manager node. Prometheus can be configured to use that information to automatically request data from all of the nodes, instead of having to manually configure each one. A new metrics_port field is added to the cluster node record that allows setting the port used by the node for metrics output.
4. A lot of the internal data we used to store about various metrics is now removed in favor of using whatever prometheus-cpp provides to us via their classes. The reasoning behind this is that we want to be able to pass the prometheus-cpp registry to external projects directly (broker/CAF being the main one) without them having to know anything about Zeek's framework itself. This lets those projects add metrics and Zeek will treat them the same as if we're add them ourselves. Our classes basically become thin wrappers around those objects, mostly providing the OpaqueVal interface to script-land plus a few other niceties.

Comments/questions above:

(2) This is a breaking change. All of the script-land options that were in the Broker module were removed, including the option change handlers, etc.
(3) This will add a management burden to users. Primarily, they need to be able to plumb whatever ports they configured for each node so that Prometheus can talk to it.
(4) This is where the majority of my questions lie:

• prometheus-cpp doesn’t keep prefix and names separately. It just stores one full, concatenated name with prefix/name/unit/total all together. This means the prefix field in the RecordVal can’t be filled in, and the name field will contain more than just the name itself. Based on the documentation from Prometheus, a metric should contain a prefix but not must contain one. I can make the assumption that all metrics in Zeek will have a prefix, scrape the first word in the name from before the first underscore, and use that as a prefix value, but that could break down if some external library does not include a prefix on one of their metrics. See the telemetry log policy for why this is a problem. It does some filtering based on the prefix values returned from Zeek.
• prometheus-cpp doesn’t keep unit information separate, so we can’t add that to the RecordVal. It's not possible to scrape this out of the full name because units may contain underscores, so it's impossible to know what set of words in a name are the unit. This page shows some samples of units, plus general naming rules for the other fields.
• prometheus-cpp doesn’t keep is_sum information separate, but we can interpret a metric name ending with _total to mean the same thing.
• prometheus-cpp only stores doubles, so I have to do a little bit of finagling with external metrics to set the right type in the RecordVal.
• prometheus-cpp doesn’t store label keys in the family, so I have to store them separately because we have other code in Zeek that validates whether the labels on a instrument match the labels on a family. For external metrics, we use the label names from the first metric in a family, making an assumption that the set of labels will always be the same for each.