telemetry: Add discovery/iteration of metrics

[awelzel]

Currently, the telemetry.bif API allows to retrieve instances of families and metrics when all parameters (prefix, name, labels, label values, etc.) are known.

For exposing metrics through script-land (e.g to create a telemetry.log), a mechanism is required to that allows the script-layer to iterate/list/discover metrics.

Brain-storming

Listing metrics API

Currently, there are 6 different helpers to create metrics. We could have 6 different listing bifs, or opt for a more union-like / &optional approach.

Current creation functions:

__int_counter_metric_get_or_add
__dbl_counter_metric_get_or_add
__int_gauge_metric_get_or_add
__dbl_gauge_metric_get_or_add
__int_histogram_metric_get_or_add
__dbl_histogram_metric_get_or_add

A single __list() bif method seems preferable.

## Generic Metric type that may be a Counter or a Gauge or a Histogram
type Metric: record {
 # opts holds prefix, name, labels, is_sum, helptext, etc.
 # This may come through the Telemetry framework. Condensing GaugeOpts/CounterOpts to MetricOpts is probably feasible and rsmmr asked about previously.
  opts: Telemetry::MetricOpts; 
  # label values in use
  label_values:  vector of string;
  # Just one of these is set.
  __int_counter_metric; opaque of int_counter_metric &optional;
  __dbl_counter_metric; opaque of int_counter_metric &optional;
  __int_gauge_metric; opaque of int_counter_metric &optional;
  __dbl_gauge_metric; opaque of int_counter_metric &optional;
  ...
  # Alternatively could maybe use higher-level Gauge/Counter types, but see int vs double discussion below.
}

function Telemetry::__list(...maybe optional filter parameters...): vector of Metric;

Translating between Metric and Gauge / Counter may be possible, but unclear if there's actually a use-case. The Metric may be mostly read-only.

Reading values: int vs double

The current script-layer proposal for a Telemetry framework opted to solely use double version as the metric data type (and hide the distinction from the user). Mostly due to Prometheus inspiration and because it makes for a simpler API.

On the C++ level, the int versions may be used and script-land consumers might be interested in the difference.

• Provide a Metric::get_value(m: Metric): double that casts the resulting value to double
• Provide a Metric::get_int_value(m: Metric) that fails if the metric isn't backed by a int version.
• Not quite applicable to histograms: These expand to multiple individual counters.

One motivating use-case may be a future telemetry.log file: If it's a simple ts, name, value structure, then value should probably be of type double. If there is int vs double differentiation, consuming the log becomes more difficult.

[awelzel]

@Neverlord - cc

[awelzel]

Some points from a discussion with @rsmmr:

• Add filter parameters to __list() for prefix, name, ...
• Provide a __list_values() specialization that also returns the current values to avoid an extra round-trip through the "value()" bifs.
• Stick with the double type, consider making count available for int counter or dbl counter that may exist in C++.
• any could be an alternative to a set of optional fields in the returned record, but unclear if better.
• Add or rename to Telemetry::list() that is visible and supported for usage by general scripts, rather than just a __list() method and assuming limited usage scope (creation of logs).

[awelzel]

I'm drafting in the telemetry branch (https://github.com/zeek/zeek/tree/topic/awelzel/telemetry-script-land-v0/scripts/base/frameworks/telemetry), but pasting a more concrete idea here.

module Telemetry;

export {
       
        ## Options type shared by all metric types
        type MetricOpts: record {
                prefix: string;
                name: string;
                ## Use the pseudo-unit "1" if this is a unit-less metric.
                unit: string;
                ## Documentation for this metric.
                helptext: string &default="Zeek Script Metric";
                labels: vector of string &default=vector();
                is_total: bool &optional;

                ## If this is a histogram, list_metrics() will populate
                ## it upon reading back, otherwise unused.
                bounds: vector of double &optional;
        };

        ## Concrete type returned when listing.
        type MetricType: enum {
                DBL_COUNTER,
                INT_COUNTER,
                DBL_GAUGE,
                INT_GAUGE,
                DBL_HISTOGRAM,
                INT_HISTOGRAM,
        };

        type ListOpts: record {
                prefix: string &default = "zeek";
                name: string &optional;

                # Maybe, filter on label values? Not sure it's worth it: Could for example
                # use gauge_with() to get a concrete Gauge instance and read it's value instead.
                # Can extend later if there's a use-case.
                # label_match: table[string] of string &optional;

                ## Set value in the returned Metric record. This
                ## only works for counter and gauges. Histograms
                ## don't have a single value.
                # It's also not clear how histograms should be
                # represented in a telemetry.log or how one
                # would work with them in a way that's easy.
                include_values: bool &optional;
        };

        ## Entry returned by :zeek:see:`Telemetry::list_metrics()`
        type Metric: record {
                opts: MetricOpts;

                ## The labels associated with this Metric.
                label_values: vector of string;

                ## Set if include_values is T. Represents the
                ## value of the gauge or counter cast to double.
                value: double &optional;

                ## Set if include_values is T and this is a int type
                ## counter or gauge. Generally prefer to just use value
                ## unless there's something that doesn't work.
                # I'm not even sure we should expose this.
                count_value: count &optional;

                ## Internal metric type.
                metric_type: MetricType;

                # References to opaque metrics. I *think* we can skip
                # them. Initially they would make it easier to read values
                # back, but can use ListOpts$include_values for that
                # and skip the complications in script-land.
                # __dbl_counter_metric: opaque of dbl_counter_metric;
                # __int_counter_metric: opaque of int_counter_metric;
                # __dbl_gauge_metric: opaque of dbl_gauge_metric;
                # __int_gauge_metric: opaque of int_gauge_metric;
        };

        ## List all existing Metric instances matching the given ListOpts
        global list_metrics: function(lo: ListOpts &default=[]): vector of Metric;

        ## Separate API to list histograms as expanded counters.
        global list_expanded_histograms: function(lo: ListOpts &default=[]): vector of Metric;
}

• ListOpts is probably not a great name, but follows the pattern
• Histograms are special, introduce a explicit list_ functions to "expand" individual counters of these.
• The MetricOptsare the same for all metrics within a family and can likely be re-used and cached, but I'm a bit uncertain how this would look on the implementation side.
• From an implementation perspective, we could probably use the existing collect() on the registry (?), but @Neverlord probably has more thoughts around this.

[awelzel]

The following PR contains the declaration and a wip implementation around __list_metrics. Should continue comments/discussion there.