Fio drop kernel cache (cloud-bulldozer#378)

bengland2 · web-flow · commit 0204b6adfb6f · 2020-09-17T08:57:29.000-04:00
* add kernel cache drop automation to ripsaw
* make kernel cache drop port configurable
* Use K8S python API to wait for DaemonSet to be in running state
* copy python script to wait for daemonset
* common code to start kernel cache dropper pod
* only invoke role when user wants kernel cache dropping
* automates starting up kernel cache dropper DaemonSet
* let benchmark-operator query nodes for labels
* pycodestyle likes this version
* move wait_for_daemonset.py to role where it is used
* add documentation for kernel cache dropper
* moved to roles/kernel_cache_drop
* fix markdown, add implementation notes and labeling
* try cache dropper in valid_fiod_hostpath.yaml
* added to CRD
* test must label cache dropper nodes
* authorize benchmark operator to query nodes
* do not set label if it is already set
* cachedrop must use benchmark-operator ServiceAccount
* Append UUID and test-name to kernel-cache-drop daemonset
* Moving update operator image to after for loop
* Update test.sh
* display security policy
* with cache dropping, tests will take longer
* try to match apiGroup name in error message
* CI fixes
diff --git a/README.md b/README.md
@@ -103,6 +103,9 @@ spec:
 ## Capturing Prometheus Data
 [Capturing Prometheus Data](docs/prometheus.md)
 
+## Cache dropping
+[Cache dropping](docs/cache_dropping.md)
+
 ## Community
 Key Members(slack_usernames): ravi, mohit, dry923, rsevilla or rook
 * [**#sig-scalability on Kubernetes Slack**](https://kubernetes.slack.com)
diff --git a/build/Dockerfile b/build/Dockerfile
@@ -8,6 +8,7 @@ RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
 COPY image_resources/centos8-appstream.repo /etc/yum.repos.d/centos8-appstream.repo
 RUN dnf install -y --nodocs redis openssl --enablerepo=centos8-appstream && dnf clean all
 
+COPY resources/kernel-cache-drop-daemonset.yaml /opt/kernel_cache_dropper/
 COPY group_vars/ ${HOME}/group_vars/
 COPY roles/ ${HOME}/roles/
 COPY templates/ ${HOME}/templates/
diff --git a/deploy/20_role.yaml b/deploy/20_role.yaml
@@ -9,6 +9,7 @@ rules:
   - ""
   resources:
   - pods
+  - daemonsets
   - services
   - endpoints
   - persistentvolumeclaims
diff --git a/deploy/40_cluster_role_kubevirt.yaml b/deploy/40_cluster_role_kubevirt.yaml
@@ -10,8 +10,23 @@ rules:
   - rolebindings
   - clusterroles
   - clusterrolebindings
+  - daemonsets
   verbs:
   - '*'
+- apiGroups:
+  - apps
+  resources:
+  - daemonsets
+  verbs:
+  - get
+  - list
+- apiGroups:
+  - rbac.authorization.k8s.io
+  resources:
+  - nodes
+  verbs:
+  - get
+  - list
 - apiGroups:
     - subresources.kubevirt.io
   resources:
diff --git a/docs/cache_dropping.md b/docs/cache_dropping.md
@@ -0,0 +1,53 @@
+This page describes how to use cache-dropping features of benchmark-operator.   
+This feature is totally optional and if you do not specify it in the CR, then it will not happen.
+
+# why drop cache
+
+Cache-dropping prevents previous state of system from affecting performance results, and this helps you
+to achieve repeatable, accurate results with low variance.
+
+Caching is an important part of any system's performance, and it is of course desirable in some cases 
+to explicitly make use of caching.   However, this cache-dropping feature does not prevent testing of
+caching performance - if you run a long enough test for the cache to "warm up", you can do this even
+with cache-dropping enabled, since cache-dropping only happens before each sample, not in the middle of
+a sample.
+
+If you want to ensure that caching will not happen during your test, you can create a data set that
+is much bigger than the amount of memory available for caching, and use a uniform random access pattern.
+
+# how to drop cache
+
+There are different types of caching that occur in the system 
+
+- kernel buffer caching
+- (Ceph OCS) OSD caching (not yet supported fully)
+
+you can control which type of cache dropping
+is done using these CR fields in the workload args section:
+
+```
+drop_cache_kernel: true
+```
+
+For this to work, you must **label** the nodes that you want to drop kernel cache, for example:
+
+```
+# kubectl label node minikube kernel-cache-dropper=yes
+```
+If you do not do this, ripsaw will timeout waiting for cache dropper pods to deploy.
+
+
+# implementation notes
+
+kernel cache dropping is done by a daemonset run on nodes with the above label.   See roles/kernel_cache_drop
+for details on how this is done.  Each pod started by this daemonset is running a CherryPy web service that
+responds to a GET URL by dropping kernel cache using equivalent of shell commnand:
+
+```
+sync 
+echo 3 > /proc/sys/vm/drop_caches
+```
+
+The sync is required because the kernel cannot drop cache on dirty pages.  
+A logfile named /tmp/dropcache.log is visible on every cache dropper pod so you can see what it's doing
+
diff --git a/group_vars/all.yml b/group_vars/all.yml
@@ -1,3 +1,4 @@
 ---
 operator_namespace: '{{ meta.namespace }}'
 clustername: 'myk8scluster'
+kernel_cache_drop_svc_port: 9222
diff --git a/playbook.yml b/playbook.yml
@@ -4,6 +4,10 @@
   gather_facts: no
   tasks:
 
+  - include_role:
+      name: "kernel_cache_drop"
+    when: workload.args.drop_cache_kernel is defined
+
   - name: Get update from Cerberus if connected
     block:
     - include_role:
diff --git a/resources/crds/ripsaw_v1alpha1_ripsaw_crd.yaml b/resources/crds/ripsaw_v1alpha1_ripsaw_crd.yaml
@@ -101,6 +101,8 @@ spec:
                 type: string
               hostpath:
                 type: string
+              drop_cache_kernel:
+                type: boolean
           status:
             type: object
             properties:
diff --git a/resources/kernel-cache-drop-clusterrole.yaml b/resources/kernel-cache-drop-clusterrole.yaml
@@ -0,0 +1,33 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: benchmark-operator
+rules:
+- apiGroups:
+  - ''
+  resources:
+  - nodes
+  - pods
+  verbs:
+  - get
+  - list
+- apiGroups:
+  - apps
+  resources:
+  - daemonsets
+  verbs:
+  - get
+  - list
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: benchmark-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: benchmark-operator
+subjects:
+- kind: ServiceAccount
+  name: benchmark-operator
+  namespace: my-ripsaw
diff --git a/resources/kernel-cache-drop-daemonset.yaml b/resources/kernel-cache-drop-daemonset.yaml
@@ -0,0 +1,42 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kernel-cache-dropper
+  namespace: my-ripsaw
+  labels:
+    app: kernel-cache-dropper
+spec:
+  selector:
+    matchLabels:
+      name: kernel-cache-dropper
+  template:
+    metadata:
+      labels:
+        name: kernel-cache-dropper
+    spec:
+      serviceAccountName: benchmark-operator
+      #tolerations:
+      #- key: node-role.kubernetes.io/master
+      #  effect: NoSchedule
+      containers:
+      - name: kernel-cache-dropper
+        image: quay.io/cloud-bulldozer/kernel_cache_dropper:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: {{ kernel_cache_drop_svc_port }}
+        env:
+            - name: KCACHE_DROP_PORT_NUM
+              value: "{{ kernel_cache_drop_svc_port }}"
+        command: ["/usr/bin/python3"]
+        args: ["/opt/kernel_cache_drop/kernel-cache-drop-websvc.py"]
+        imagePullPolicy: Always
+        securityContext:
+          privileged: true
+        # we don't need all the same volumes as the toolbox pod
+        volumeMounts:
+        - name: proc-sys-vm
+          mountPath: /proc_sys_vm
+      volumes:
+      - name: proc-sys-vm
+        hostPath:
+          path: /proc/sys/vm
diff --git a/roles/fio_distributed/templates/client.yaml b/roles/fio_distributed/templates/client.yaml
@@ -27,6 +27,10 @@ spec:
             value: "{{ clustername }}"
           - name: ceph_cache_drop_pod_ip
             value: "{{ workload_args.rook_ceph_drop_cache_pod_ip | default() }}"
+          - name: kcache_drop_pod_ips
+            value: "{{ kcache_drop_pod_ips | default() }}"
+          - name: KCACHE_DROP_PORT_NUM
+            value: "{{ kernel_cache_drop_svc_port }}"
 {% if elasticsearch is defined %}
           - name: es
             value: "{{ elasticsearch.server }}"
diff --git a/roles/kernel_cache_drop/tasks/main.yml b/roles/kernel_cache_drop/tasks/main.yml
@@ -0,0 +1,32 @@
+---
+- debug:
+    msg: "in cache dropper role"
+
+- name: start kernel cache dropper
+  k8s:
+    definition: "{{ lookup('template', '/opt/kernel_cache_dropper/kernel-cache-drop-daemonset.yaml') | from_yaml }}"
+
+- name: get pod IPs of running DaemonSet
+  shell: "python3 /opt/ansible/roles/kernel_cache_drop/wait_for_daemonset.py 30 {{ operator_namespace }} kernel-cache-dropper"
+
+- name: get kernel cache dropper pods
+  k8s_facts:
+    kind: Pod
+    label_selectors:
+      - name = kernel-cache-dropper
+    namespace: "{{ operator_namespace }}"
+  register: kcache_drop_pod_list
+
+- debug:
+    var: kcache_drop_pod_list
+
+- name: put ip list into a var
+  set_fact:
+    kcache_drop_pod_ips: |
+       {% for item in kcache_drop_pod_list.resources %}
+       {{ item.status.podIP }}
+       {% endfor %}
+
+- debug:
+    var: kcache_drop_pod_ips
+
diff --git a/roles/kernel_cache_drop/wait_for_daemonset.py b/roles/kernel_cache_drop/wait_for_daemonset.py
@@ -0,0 +1,86 @@
+#!/usr/bin/python3
+# wait_for_daemonset.py -
+# wait until all pods are in Running state for specified DaemonSet
+# see usage() for input params
+
+from kubernetes import client, config
+from sys import argv, exit
+import time
+
+NOTOK = 1
+# may have to adjust this based on number of pods
+poll_interval = 2.0
+
+
+def usage(msg):
+    print('ERROR: %s' % msg)
+    print('usage: wait_for_daemonset.py timeout namespace node-label')
+    exit(NOTOK)
+
+
+# parse command line
+
+if len(argv) < 3:
+    usage('too few parameters')
+
+timeout_str = argv[1]
+ns = argv[2]
+label = argv[3]
+
+# show 'em what we parsed
+
+print('timeout if pods not seen in %s sec' % timeout_str)
+print('namespace: %s' % ns)
+print('node label: %s' % label)
+
+timeout = int(timeout_str)
+if timeout <= poll_interval:
+    usage('timeout %d must be greater than poll interval %d' %
+          (timeout, poll_interval))
+
+# wait for pods
+
+# cannot do this from inside cluster pod: config.load_kube_config()
+config.load_incluster_config()
+v1 = client.CoreV1Api()
+
+nodes = v1.list_node().items
+matching_nodes = 0
+for n in nodes:
+    try:
+        v = n.metadata.labels[label]
+        matching_nodes += 1
+    except KeyError:
+        pass
+
+print('expecting %d nodes to have this pod' % matching_nodes)
+if matching_nodes == 0:
+    usage('at least 1 node must have the %s label' % label)
+
+print('waiting for daemonset...')
+start_time = time.time()
+matching_pods = 0
+generate_name = label + '-'
+
+while True:
+    time.sleep(poll_interval)
+    now_time = time.time()
+    delta_time = now_time - start_time
+    if delta_time > timeout:
+        break
+
+    ret = v1.list_pod_for_all_namespaces(watch=False)
+    matching_pods = 0
+    for i in ret.items:
+        if i.metadata.namespace == ns and \
+           i.metadata.generate_name == generate_name and \
+           i.status.phase == 'Running':
+            matching_pods += 1
+    if matching_pods >= matching_nodes:
+        break
+
+if delta_time > timeout:
+    usage('timeout waiting for pods to reach running state')
+
+if matching_pods != matching_nodes:
+    usage('expected %d pods, found %d pods' % (matching_nodes, matching_pods))
diff --git a/test.sh b/test.sh
@@ -66,6 +66,9 @@ mkdir gold
 NEW_UUID=$(uuidgen)
 UUID=${NEW_UUID%-*}
 
+#Tag name
+tag_name="${NODE_NAME:-master}"
+
 sed -i "s/ES_SERVER/$ES_SERVER/g" tests/test_crs/*
 sed -i "s/ES_PORT/$ES_PORT/g" tests/test_crs/*
 sed -i "s/sql-server/sql-server-$UUID/g" tests/mssql.yaml tests/test_crs/valid_hammerdb.yaml tests/test_hammerdb.sh
@@ -79,10 +82,9 @@ sed -i "s/kind: Benchmark/kind: Benchmark-$UUID/g" tests/test_crs/*.yaml
 sed -i "s/kind: Benchmark/kind: Benchmark-$UUID/g" playbook.yml
 sed -i "s/kind: Benchmark/kind: Benchmark-$UUID/g" watches.yaml
 sed -i "s/backpack_role/backpack_role-$UUID/g" resources/backpack_role.yaml
+sed -i "s/my-ripsaw/my-ripsaw-$UUID-test-fiod/g" resources/kernel-cache-drop-daemonset.yaml
 grep -Rl "kind: Benchmark" roles/ | xargs sed -i "s/kind: Benchmark/kind: Benchmark-$UUID/g"
-
-# Update the operator image
-update_operator_image
+sed -i "s|          image: quay.io/benchmark-operator/benchmark-operator:master*|          image: $image_location/$image_account/benchmark-operator:$tag_name # |" resources/operator.yaml
 
 cp -pr * gold/
 
@@ -94,9 +96,13 @@ do
   cd $ci_dir/
   # Edit the namespaces so we can run in parallel
   sed -i "s/my-ripsaw/my-ripsaw-$UUID-$ci_dir/g" `grep -Rl my-ripsaw`
+  sed -i "s/benchmark-operator-role/benchmark-operator-role-$UUID-$ci_dir/g" `grep -Rl benchmark-operator-role`
   cd ..
 done
 
+# Update the operator image
+update_operator_image
+
 # Run scale test first if it is in the test list
 scale_test="false"
 if [[ `grep test-scale-openshift tests/my_tests` ]]
diff --git a/tests/common.sh b/tests/common.sh
@@ -162,6 +162,14 @@ function operator_requirements {
   kubectl apply -f resources/namespace.yaml
   kubectl apply -f deploy
   kubectl apply -f resources/crds/ripsaw_v1alpha1_ripsaw_crd.yaml
+  kubectl -n my-ripsaw get roles
+  kubectl -n my-ripsaw get rolebindings
+  kubectl -n my-ripsaw get podsecuritypolicies
+  kubectl -n my-ripsaw get serviceaccounts
+  kubectl -n my-ripsaw get serviceaccount benchmark-operator -o yaml
+  kubectl -n my-ripsaw get role benchmark-operator -o yaml
+  kubectl -n my-ripsaw get rolebinding benchmark-operator -o yaml
+  kubectl -n my-ripsaw get podsecuritypolicy privileged -o yaml
 }
 
 function backpack_requirements {
diff --git a/tests/test_crs/valid_fiod_hostpath.yaml b/tests/test_crs/valid_fiod_hostpath.yaml
@@ -15,6 +15,7 @@ spec:
   workload:
     name: "fio_distributed"
     args:
+      drop_cache_kernel: true
       samples: 1
       servers: 2
       prefill: true
diff --git a/tests/test_fiod.sh b/tests/test_fiod.sh
