metrics.init() method calls metrics.Register() of client-go, cause other component can not observe metrics of client-go

[shaofeng66]

https://github.com/kubernetes/client-go/blob/master/tools/metrics/metrics.go#L132

metrics.Register() is in a sync.Once.Do(), so the register can only has effect once.

My program import controller runtime to use its client, and I want to track the latency of all client-go requests and other metrics defined in client-go. But I can not register any other metrics to client-go b/c contoller-runtime has done it with a single "RequestResult".

Or is there any other method that I can use to get the metrics from client-go?

[shaofeng66]

An W.A. would be call my own registring code before "importing" controller runtime, by adding some code like

import _ "github.com/myorg/myproj/pkg/metrics"

in main.go before any controller runtime initilaztion.

But I think an explicit named initilaztion func and it invokation is better than the init()?

[f41gh7]

As a workaround, It should be possible provide own adapter to the client-go/metrics package and use it with metrics collector.

import (
	"github.com/prometheus/client_golang/prometheus"
  	 restmetrics "k8s.io/client-go/tools/metrics"
)
var requestLatencySeconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{Name: "rest_client_request_duration_seconds"}, []string{"verb", "path"})

func exposeRestClientMetrics(r *prometheus.Registry){
  r.MustRegister(requestLatencySeconds)
  restmetrics.RequestLatency = &latencyAdapter{collector: requestLatencySeconds}
}

type latencyAdapter struct {
	collector *prometheus.HistorgramVec
}

func (la *latencyAdapter) Observe(ctx context.Context, verb string, u url.URL, latency time.Duration) {
	la.collector.WithLabelValues(verb, u.Path).Observe(latency.Seconds())
}

[alvaroaleman]

But I think an explicit named initilaztion func and it invokation is better than the init()?

The issue is that this means it will break for the majority of users who don't want to do anything and just get metrics by default. Retaining both that and avoiding this issue will be difficult. IMHO the best solution for this would be for upstream not to make this a sync.Once.

We could change downstream to register them right before first use rather than in an init, that would slightly improve things in that you can just register them before constructing controllers, but it is still a bit arcane and not obvious that it will not work if you register them too late.

I opened kubernetes/kubernetes#127739 in upstream, because I don't think its possible to provide a "good" solution for this downstream.