Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to enable loopback feature on IBM Cloud Openshift #595

Open
ocofaigh opened this issue Jan 13, 2024 · 1 comment
Open

Unable to enable loopback feature on IBM Cloud Openshift #595

ocofaigh opened this issue Jan 13, 2024 · 1 comment

Comments

@ocofaigh
Copy link

ocofaigh commented Jan 13, 2024

Environment: IBM Cloud Openshift cluster 4.13

Steps:

  1. git clone [email protected]:logdna/logdna-agent-v2.git
  2. cd logdna-agent-v2
  3. git checkout 3.9.1
  4. oc new-project logdna-agent
  5. oc create serviceaccount logdna-agent
  6. oc create secret generic logdna-agent-key --from-literal=logdna-agent-key=XXXXXX
  7. oc adm policy add-scc-to-user privileged system:serviceaccount:logdna-agent:logdna-agent
  8. Modified the yaml k8s/agent-resources-openshift.yaml with the following changes:

Problem:
The initContainer fails with:

% oc logs logdna-agent-6gkdq -c volume-mount-permissions-fix
chmod: /var/lib/logdna: Permission denied
chmod: /var/lib/logdna: Permission denied

Why would the initContainer running as root with the privileged SCC not be able to set permissions on /var/lib/logdna?

I even exec into the initContiner, and can see this:

/ # whoami
root
/ # id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
/ # ls -la /var/lib/logdna/
total 12
drwxr-xr-x    2 root     root          4096 Jan 13 20:25 .
drwxr-xr-x    3 root     root          4096 Jan 13 20:38 ..

Here is the final yaml I used:

---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: logdna-agent-ds-priority
  namespace: logdna-agent
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Logdna Agent"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: logdna-agent
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get","list", "create", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: logdna-agent
  namespace: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: logdna-agent
subjects:
  - kind: ServiceAccount
    name: logdna-agent
    namespace: logdna-agent
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get","list", "create", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: logdna-agent
subjects:
  - kind: ServiceAccount
    name: logdna-agent
    namespace: logdna-agent
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logdna-agent
  namespace: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 100%
  selector:
    matchLabels:
      app: logdna-agent
  template:
    metadata:
      labels:
        app: logdna-agent
        app.kubernetes.io/name: logdna-agent
        app.kubernetes.io/instance: logdna-agent
        app.kubernetes.io/version: 3.9.1
    spec:
      serviceAccountName: logdna-agent
      priorityClassName: logdna-agent-ds-priority
      initContainers:
        - name: volume-mount-permissions-fix
          image: busybox
          command: ["sh", "-c", "chmod -R 775 /var/lib/logdna && chown -R 5000:5000 /var/lib/logdna"]
          volumeMounts:
          - name: varliblogdna
            mountPath: /var/lib/logdna
      containers:
        - name: logdna-agent
          image: logdna/logdna-agent:3.9.1
          imagePullPolicy: Always
          securityContext:
            runAsUser: 5000
            runAsGroup: 5000
            privileged: true
            capabilities:
              add:
                - DAC_READ_SEARCH
              drop:
                - all
          env:
            - name: LOGDNA_INGESTION_KEY
              valueFrom:
                secretKeyRef:
                  name: logdna-agent-key
                  key: logdna-agent-key
            - name: LOGDNA_LOOKBACK
              value: smallfiles
            - name: LOGDNA_DB_PATH
              value: /var/lib/logdna
            - name: POD_APP_LABEL
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['app.kubernetes.io/name']
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 20m
            limits:
              memory: 500Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: vardata
              mountPath: /var/data
            - name: varliblogdna
              mountPath: /var/lib/logdna
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: mnt
              mountPath: /mnt
              readOnly: true
            - name: osrelease
              mountPath: /etc/os-release
            - name: logdnahostname
              mountPath: /etc/logdna-hostname
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: vardata
          hostPath:
            path: /var/data
        - name: varliblogdna
          hostPath:
            path: /var/lib/logdna
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: mnt
          hostPath:
            path: /mnt
        - name: osrelease
          hostPath:
            path: /etc/os-release
        - name: logdnahostname
          hostPath:
            path: /etc/hostname
@ocofaigh
Copy link
Author

Oh, I found the issue. I had to explicitly add this to the initContainer:

securityContext:
  privileged: true
  runAsUser: 0

This is missing from the doc here -> https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md#enabling-file-offset-tracking-across-restarts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant