Metrics support

[jean-airoldie]

I was about to start working on a small crate to export procfs via prometheus using metrics until I saw your crate. Would you be interested in supporting metrics? And if so, how can I help?

[svartalf]

Hey, @jean-airoldie! I was experimenting with the metrics crate and Prometheus export too, you can find the sources here, it is broken a little bit (has this issue with counters and absolute values), but fits okay as a proof of concept.

I'm inclined not to integrate metrics into the heim for many reasons:

1. it would increase code complexity drastically, adding to maintenance burden
2. it will lead to extra memory allocations; metrics requires either owned Strings or static &str for label pairs, but heim API usually returns references or types, which should be converted into a String manually, ex. &Path or &OsStr
3. should it expose platform-specific data too?
4. exposed metric names might not suit everyone needs

On the other hand, we could take this heim-node-exporter, re-brand it maybe and export everything what heim can provide, if you want to join the idea. How does that sound to you?

[jean-airoldie]

Yep, that sounds good. The general ideas I had in my mind:

• Make a node exporter builder used to configure what metrics are exposed, etc.
• We run the exporter as a metrics proxy. This means that we will only run heim when a snapshot is requested (e.g. every 15sec with prometheus).
• We could also add a feature that register the proxy automatically with the default settings,
  so that a user could simply depend on heim-node-exporter to get access to these metrics (no config needed).

[svartalf]

Okay, first of all, I moved this issue into the heim-rs/heim-node-exporter repo.

Make a node exporter builder used to configure what metrics are exposed, etc.

Sounds good, same as Prometheus' node_exporter does: https://github.com/prometheus/node_exporter#collectors

We run the exporter as a metrics proxy. This means that we will only run heim when a snapshot is requested (e.g. every 15sec with prometheus).
We could also add a feature that register the proxy automatically with the default settings,
so that a user could simply depend on heim-node-exporter to get access to these metrics (no config needed).

I do not think that I follow this part. At the moment heim-node-exporter acts as a HTTP server, which exposes metrics via HTTP endpoint for scrape_config later and the actual information fetch happens during the HTTP request processing.

Why would it be necessary to "proxy" it via the metrics proxy? Also it seems that it adds the common prefix for all metrics, not sure if it is reasonable and even compatible with Prometheus naming format at all.

[jean-airoldie]

Currently node export is a binary that must be run to expose the desired metrics. It works when the only thing you want to do is export heim metrics. However, this is not scalable in a container setting.

Usually in a container you only want to run one binary, your application. However with this design you would have to fork an additional heim-node-exporter process in the background. Moreover, you would need to parse 2 different HTTP servers, one for heim-node-exporter and one for your application (assuming your are exposing metrics in your app). This complicates the prometheus config.

What I propose instead is to use a proxy so that we can register the collectors::collect() call to be run each time a snapshot is requested (e.g. by a request to the HTTP server). This means that the metrics from heim-node-exporter will be automatically exported to any library that depends on it. In other words, we want to register the heim-node-exporter to use the same global Recorder that the application uses so that the http-exporter thread will also export heim metrics.

[jean-airoldie]

Also it seems that it adds the common prefix for all metrics, not sure if it is reasonable and even compatible with Prometheus naming format at all.

I'll look into it.

[jean-airoldie]

The ideal scenario for me would be something like:

[dependencies]
heim-node-exporter = { version = "0.0.1", features = ['install'] }

And no additionnal configuration needed.

[svartalf]

Oh, okay, I got it now, it seems reasonable solution for containers. But now it seems that it does not belong to heim-node-exporter, as I want it to stay as a standalone HTTP server :)

What would you say if I create heim-metrics-proxy crate and we would continue with it? Or, you can jump into the gitter chat, so we could discuss it at there