Custom label for the client service, to scrape metrics from the etcd cluster pods

[sanamsarath]

The problem description

Trying to deploy etcd cluster with Prometheus. Since the Prometheus service monitor rely on the label selector of the services to scrap the metrics from the etcd pods, having client and peer services with same label is causing the Prometheus to scrape the metrics from each etcd pod 2 times.

e.g.
etcd client and peer services:

[kuberadmin@sarath-reddy-k8-master-1-a00031-4b2c09f7b029c1b8 ~]$ kc get svc -o wide
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE   SELECTOR
etcd-cluster          ClusterIP   None            <none>        2379/TCP,2380/TCP   31m   app=etcd,etcd_cluster=etcd-cluster
etcd-cluster-client   ClusterIP   10.110.65.249   <none>        2379/TCP            31m   app=etcd,etcd_cluster=etcd-cluster

my servicemonitor :

[kuberadmin@sarath-reddy-k8-master-1-a00031-4b2c09f7b029c1b8 ~]$ kc describe ServiceMonitor
Name:         fed-db-etcd-etcd-cluster-monitor
Namespace:    fed-db-etcd
Labels:       app=etcd-cluster
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Creation Timestamp:  2019-10-30T02:36:28Z
  Generation:          1
  Resource Version:    14992039
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/fed-db-etcd/servicemonitors/fed-db-etcd-etcd-cluster-monitor
  UID:                 138816e9-fabe-11e9-ae79-fa163e23c1fc
Spec:
  Endpoints:
    Interval:  5s
    Port:      client
  Job Label:   etcd-cluster
  Namespace Selector:
    Match Names:
      fed-db-etcd
  Selector:
    App:  etcd
Events:   <none>

Since both the services are defined with the label app=etcd and both has port name client, prometheus is running metrics on endpoints of both services etcd-cluster and etcd-cluster-client.

Need suggestion
Assuming people are already using Prometheus with etcd-operator and etcd-cluster, is there any solution to define ServiceMonitor so that Prometheus can only select "etcd-cluster-client" service.

[sanamsarath]

@hexfusion any suggestion??

-Thanks.