TestPMD is an application used to test the DPDK in a packet forwarding mode, here this application run inside a pod with SRIOV VF.
This workload assumes a healthy cluster is runing with SR-IOV and Performance Add-On operators.
- SR-IOV operator is required to create/assign VFs to the pods
- PAO is required to prepare the worker node with Hugepages for DPDK application and isolate CPUs for workloads.
These are used to create VFs and SRIOV networks, interface might vary. Creating SriovNetworkNodePolicy will reboot your worker nodes one by one.
Below node policy is for Intel XXV710 NICs, for a Mellanox NIC to work in Data Plane Development Kit (DPDK) mode, use the netdevice driver type and include additional spec isRdma: true.
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: testpmd-policy
namespace: openshift-sriov-network-operator
spec:
deviceType: vfio-pci
nicSelector:
pfNames:
- ens2f1
nodeSelector:
node-role.kubernetes.io/worker: ""
numVfs: 20
resourceName: intelnicsapiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: testpmd-sriov-network
namespace: openshift-sriov-network-operator
spec:
ipam: |
{
"type": "host-local",
"subnet": "10.57.1.0/24",
"rangeStart": "10.57.1.100",
"rangeEnd": "10.57.1.200",
"routes": [{
"dst": "0.0.0.0/0"
}],
"gateway": "10.57.1.1"
}
spoofChk: "on"
trust: "on"
resourceName: intelnics
networkNamespace: benchmark-operator
The CPU and socket information might vary, use the below as an example. Creating PerformanceProfile will trigger MCO to reboot worker nodes one by one.
Please be sure to WAIT until the previous step is finished, i.e SR-IOV VF configuration.
Add label to the MCP pool before creating the performance profile.
# oc label mcp worker machineconfiguration.openshift.io/role=worker---
apiVersion: performance.openshift.io/v1
kind: PerformanceProfile
metadata:
name: testpmd-performance-profile-0
spec:
cpu:
isolated: "6,8,10,12,14,16,18,20,22,24,26,28"
reserved: "0-5,7,9,11,13,15,17,19,21,23,25,27,29"
hugepages:
defaultHugepagesSize: "1G"
pages:
- size: "1G"
count: 16
node: 0
numa:
topologyPolicy: "best-effort"
nodeSelector:
node-role.kubernetes.io/worker: ""
realTimeKernel:
enabled: false
additionalKernelArgs:
- nosmt
- tsc=reliableA minimum configuration to run testpmd and trex with in pod,
apiVersion: ripsaw.cloudbulldozer.io/v1alpha1
kind: Benchmark
metadata:
name: testpmd-benchmark
namespace: benchmark-operator
spec:
clustername: myk8scluster
workload:
name: testpmd
args:
privileged: true
pin: false
pin_testpmd: "node-0"
pin_trex: "node-0"
networks:
testpmd:
- name: testpmd-sriov-network
count: 2 # Interface count, Min 2
trex:
- name: testpmd-sriov-network
count: 2Additional testpmd, trex and pod specification can also be supplied in CR definition yaml(below specs are current defaults)
Pod specification arguments,
workload:
...
pod_hugepage_1gb_count: 4Gi
pod_memory: "1000Mi"
pod_cpu: 6TestPMD arguments for more description refer here
workload:
...
socket_memory: 1024,0 # This memory is allocated based on your node hugepage allocation, ex. for NUMA 0.
memory_channels: 4
forwarding_cores: 4
rx_queues: 1
tx_queues: 1
rx_descriptors: 1024
tx_descriptors: 1024
forward_mode: "mac"
stats_period: 1
disable_rss: true TRex arguments
workload:
...
duration: 30 # In Seconds
packet_size: 64
packet_rate: "10kpps" # All lower case and only support Packets per seconds
num_stream: 1TestPMD Pod uses this image
it can be overriden by supplying additional inputs in CR definition yaml, provided testpmd command will be executable from the image WORKDIR with additional environment variables
workload:
...
image_testpmd: "registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6"
image_pull_policy: "Always"
environments:
testpmd_mode: "direct" ## Just an exampleTRex Pod uses snafu based image
This CR will create TestPMD pod and TRex pod using the SRIOV network provided and run traffic between them. The result can be viewed as below,
# oc get benchmark
NAME TYPE STATE METADATA STATE UUID AGE
testpmd-benchmark testpmd Complete not collected 35daf5ac-2edf-5e34-a6cc-17fcda055937 4m36s
# oc get pods
NAME READY STATUS RESTARTS AGE
benchmark-operator-7fff779945-d9485 3/3 Running 0 4m16s
testpmd-application-pod-35daf5ac-nxc4v 1/1 Running 0 2m41s
trex-traffic-gen-pod-35daf5ac-dslzh 0/1 Completed 0 2m20s
oc logs trex-traffic-gen-pod-35daf5ac-dslzh
2021-01-28T17:41:02Z - INFO - MainProcess - run_snafu: logging level is INFO
2021-01-28T17:41:02Z - INFO - MainProcess - wrapper_factory: identified trex as the benchmark wrapper
2021-01-28T17:41:02Z - INFO - MainProcess - trigger_trex: Starting TRex Traffic Generator..
2021-01-28T17:41:47Z - INFO - MainProcess - trigger_trex: {
"cluster_name": "myk8scluster",
"cpu_util": 0.22244608402252197,
"duration": 30,
"kind": "pod",
"num_stream": 1,
"packet_rate": "10kpps",
"packet_size": 64,
"rx_bps": 17426300.0,
"rx_drop_bps": 0.0,
"rx_pps": 19984.2890625,
"testpmd_node": "worker000-fc640",
"trex_node": "worker001-fc640",
"tx_bps": 16786740.0,
"tx_pps": 19984.212890625,
"user": "snafu",
"uuid": "35daf5ac",
"workload": "testpmd"
}
2021-01-28T17:41:47Z - INFO - MainProcess - trigger_trex: Finished Generating traffic..
2021-01-28T17:41:47Z - INFO - MainProcess - run_snafu: Duration of execution - 0:00:45, with total size of 240 bytesTo publish data to Elasticsearch, the procedure is same as other workloads.
spec:
clustername: testpmd
test_user: snafu
elasticsearch:
url: https://search-perfscale-dev.server.com:443For Long running tasks, its better to set the stats collection interval which would limit the amount of data it collects. By default script collects data at every second, this setting would let user to change the resolution.
workload:
name: testpmd
args:
stats_interval: 10 # Cannot be less than 1 or a float object