Support for Istio prometheusMerge

[a-thaler]

Description
Dynatrace metric integration is cumbersome with metrics as there is not the best fitting solution available. Either the user disabled istio mTLS for the metric ports and uses dynatrace annotations, or he uses the telemetry module to push data via OTLP to Dynatrace, but hear so many conversion is needed that it is troublefull as well, also see https://kyma-project.io/#/telemetry-manager/user/integration/dynatrace/README?id=ingest-metrics

It turned out that there is another alternative to the direct Dynatrace integration scenario which does not require an explicit disabling of the TLS mode for the whole port. By enabling the prometheusMerge feature of Istio, the envoys will expose a new port which serves the envoy metrics plus the business metrics (configured via prometheus pod annotation). With that, Dynatrace can scrape all the metrics directly without reducing the TLS setting on the original port. However, the metric data will be transported plain text.

While that solution sounds better then the current integration scenario, it will affect the current telemetry functionality heavily, as suddenly all pods will have prometheus annotations and will be detected by the telemetry prometheus input. And all this pods will mainly return envoy metrics.

Goal
Understand all the implications of the feature and propose a concept on if the istio feature should be enabled always/optional and if something should be adjusted in the telemetry functionality.

Criterias

• A new document is created in the internal docs folder which:
  • gives a clear proposal on if the istio feature should be enabled by default/optional
  • what kind of adjustments are needed on the telemetry functionality