5. log collection

log collection

Log collection will be performed using the fluent-operator component

deploy fluent-operator

docker

helm install fluent-operator --create-namespace -n infra \
--set containerRuntime=docker \
--set fluentbit.enable=false \
--set fluentbit.input.tail.enable=false \
--set fluentbit.input.systemd.enable=false \
--set fluentbit.filter.kubernetes.enable=false \
--set fluentbit.filter.containerd.enable=false \
--set fluentbit.filter.systemd.enable=false \
https://github.com/fluent/fluent-operator/releases/download/v2.7.0/fluent-operator.tgz

containerd or cri-o

helm install fluent-operator --create-namespace -n infra \
--set containerRuntime=containerd \
--set fluentbit.enable=false \
--set fluentbit.input.tail.enable=false \
--set fluentbit.input.systemd.enable=false \
--set fluentbit.filter.kubernetes.enable=false \
--set fluentbit.filter.containerd.enable=false \
--set fluentbit.filter.systemd.enable=false \
https://github.com/fluent/fluent-operator/releases/download/v2.7.0/fluent-operator.tgz

configure logging

Modify the actual parameters for clusterName and es.host in the configuration file.

• https://github.com/buxiaomo/fluent-operator.git

git clone https://github.com/buxiaomo/fluent-operator.git
cd fluent-operator
kubectl apply -f . -n infra

index rule

• devops-kube-audit include kubernetes audit

• devops-kube-event include kubernetes event

• devops-kube-component include systemd log(containerd、crio、docker、etcd、haproxy、keepalived、kube-apiserver、kube-controller-manager、kube-proxy、kube-scheduler、kubelet)

• devops-applog-kube-system include kube-system namespace pod log

• devops-applog-infra include infra namespace pod log

• devops-applog-<projectname>_(prod|noprod) include project namespace Pod log

example:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: containers-<namespace>
  labels:
    fluentbit.fluent.io/infra: "true"
spec:
  matchRegex: kube.var.log.containers.*_<namespace>_.*-(\w|\.){68}$
  es:
    host: 172.16.100.10
    index: devops-applog-<projectname>-(prod|noprod)-%Y.%m.%d
    port: 9200
    replaceDots: true
    suppressTypeName: "false"
    traceError: true
    type: _doc
    bufferSize: 10M