Transform & Filter support for OTel pipelines

[a-thaler]

Description
Users like the way it is by getting a kind of managed agent support to easily start collecting telemetry data. The usual integration with CloudLogging seem to fit even from the requirement perspective, however, as soon as a custom output gets used, custom filters for LogPipelines are getting used. Also, often I wanted to recommend to filter out some metrics coming with the default setup, where such possibility is missing at the moment.

Requirements:

• Have a way to enrich data with cluster specific data attributes
• Have a way to filter out data which is not relevant for the use case
• The support should be equivalent across all Otel based pipelines, using similar syntax and semantics

Idea:

• All generic filter and transformation logic in otel-collector-contrib seem to get consolidated into one main processors, filterprocessor and transformprocessor, being both based on OTTL.
• Provide support for these two processors, and all major needs in regards to filter and transform will be covered

Typical Use cases:

• Add one cluster identifier resource attribute like for dynatrace
• Do typical transformations as needed for dynatrace, see integration guide values.yaml
• Add proper span.status in case of 4xx responses
• filter out metrics by name pattern like "k8s.node.*"

Tasks:

• Preparation
  • Understand the OTTL package and decide if it can be used as external interface #1483
  • Come up with an API proposal
    - Do you need to have multiple filter and transform rules with a flexible order, or is one block for each in a fixed order is enough?
    - Can it be on root level or do we need an additional umbrella attribute "processors"?
• Implementation