help request: Fluentd Prometheus Plugin and Metrics Expose

[vajgi90]

Describe the issue

Dear Community,

I would like my Fluentd pods to provide metrics about themselves on 24231 port /metrics patch.

It's not entirely clear to me, but I wanted this block

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

to be included in my fluentd config and the metrics to be placed in the /metrics path.

To do this, I also created a ClusterInput:

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterInput
metadata:
  labels:
    config.fluentd.fluent.io/enabled: 'true'
  name: prometheus-metrics
spec:
  inputs:
    - customPlugin:
        config: |
          <source>
            @type prometheus
            bind 0.0.0.0
            port 24231
            metrics_path /metrics
          </source>

And I patched: ClusterFluentdConfig with this because by default the template does not contain it:

clusterInputSelector:
matchLabels:
filter.fluentd.fluent.io/enabled: "true"

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  labels:
    config.fluentd.fluent.io/enabled: "true"
  name: fluentd-config
spec:
  clusterFilterSelector:
    matchLabels:
      filter.fluentd.fluent.io/enabled: "true"
  clusterInputSelector:
    matchLabels:
      filter.fluentd.fluent.io/enabled: "true"
  clusterOutputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"
  watchedNamespaces: []

I still find that the Prometheus-specific Source is not included in the configuration (fluentd-config k8s secret).

Do you have any tips on what else might be missing? Thank you in advance for your help.

How did you install fluent operator?

With Helm Chart ( version 3.3.0).

Additional context

No response

[snoop123]

@vajgi90 You are missing the input label selector in your ClusterInput. This worked for me (I used input.fluentd.fluent.io/enabled as ClusterInputSelector):

kind: ClusterInput
apiVersion: fluentd.fluent.io/v1alpha1
metadata:
  name: monitor-agent
  labels:
    input.fluentd.fluent.io/enabled: "true"
spec:
  inputs:
    - monitorAgent:
        bind: 0.0.0.0
        port: 24220

Also created PR to address this ClusterInputSelector addition
fluent/helm-charts#669