add exporter runtime metrics

[pryorda]

From @wtip on September 12, 2017 21:17

It would be nice to add some exporter runtime metrics.
For example at the moment it would be really useful if I knew how long the exporter is taking to scrape vmware for metrics.

Copied from original issue: rverchere/vmware_exporter#17

[pryorda]

@wtip Are the logs enough or do we want it to be exposed as an actual metric? I have no real preference here.

[wtip]

If you don't expose it as a prometheus metric you can't monitor the scrape duration with prometheus.
I'm in favor of having an actual metric.

[dannyk81]

@wtip actually, scrape duration are automatically exposed as metrics by prometheus itself (scrape_duration_seconds metric) you just need to make sure your prometheus server scrapes its own metrics.

Other "self" metrics could be relevant to monitor things like cpu usage, memory usage, threads, etc... the prometheus python_client should be able to provide all that.

[brian-brazil]

you just need to make sure your prometheus server scrapes its own metrics.

That's not required for scrape_duration_seconds, it's put into the TSDB as part of the scrape just like up.

[dannyk81]

@brian-brazil you're right! even better 💯

[pryorda]

@brian-brazil++

[finkr]

This exporter actually has multiple collectors running in parallel, it would be nice to have metrics to track how each one is behaving. node_exporter is doing that (see node_scrape_collector_duration_seconds at demo.robustperception.io ).

Also, the python_client metrics aren't available yet.

[finkr]

Instead of the globabl registry, we could use an alternate endpoint for internal metrics. usualy

from prometheus_client.twisted import MetricsResource
[..]
root.putChild(b'metrics', VMWareMetricsResource(args))
root.putChild(b'healthz', HealthzResource())
root.putChild(b'metrics2', MetricsResource())