Skip to content

The QoS project is a cloud-native solution leveraging eBPF technology, designed to efficiently manage and optimize network traffic across diverse hybrid deployment scenarios.

License

Notifications You must be signed in to change notification settings

AliyunContainerService/terway-qos

Repository files navigation

qos

Introduction

terway-qos is developed to solve the problem of container network bandwidth contention in mixed deployment scenarios. It supports bandwidth limitation based on individual Pods and business types.

Compared to other solutions, terway-qos has the following advantages:

  1. Supports bandwidth limitation based on business types, allowing for mixed deployment of multiple business types.
  2. Supports dynamic adjustment of Pod bandwidth limitation.

Functionality

Bandwidth limitation can be divided into:

  • Host bandwidth limitation
  • Pod bandwidth limitation

Host bandwidth limitation

In mixed deployment scenarios, we expect to guarantee maximum bandwidth for online business to avoid contention. During idle periods, offline business should also be able to utilize the full bandwidth resources as much as possible. For this purpose, users can define three priority levels for business traffic: L0, L1, and L2. The priority order is L0 > L1 > L2.

Definition of contention scenario: When the total traffic of L0, L1, and L2 exceeds the host bandwidth.

Limitation strategy:

  • The maximum bandwidth of L0 is dynamically adjusted based on the real-time traffic of L1 and L2. The maximum value is the host bandwidth, and the minimum value is host bandwidth - minimum L1 bandwidth - minimum L2 bandwidth.
  • Under any circumstances, the bandwidth of L1 and L2 should not exceed their respective upper limits.
  • In a contention scenario, the bandwidth of L1 and L2 should not be lower than their respective lower limits.
  • In a contention scenario, the bandwidth is limited in the order of L2, L1, and L0.

Supports hot update of annotations to adjust Pod bandwidth limitation.

Please note that the CNI plugin may also support Kubernetes standard annotations, which may affect the hot update. In this case, you can choose to disable the bandwidth limitation feature of the CNI plugin.

Pod priority definition

Configure the following annotation for Pods:

key Parameters
k8s.aliyun.com/qos-class guaranteed for online business L0
burstable for offline business L1
best-effort for offline business L2

Bandwidth limitation configuration

For nodes requiring mixed deployment, configure the grace limits in the path /var/lib/terway/qos/global_bps_config.

Configuration Path Parameters
/var/lib/terway/qos/global_bps_config hw_tx_bps_max maximum tx bandwidth for the node
hw_rx_bps_max maximum rx bandwidth for the node
offline_l1_tx_bps_min minimum guaranteed bandwidth for inbound L1 offline business
offline_l1_tx_bps_max maximum bandwidth usage for inbound L1 offline business
offline_l2_tx_bps_min minimum guaranteed bandwidth for inbound L2 offline business
offline_l2_tx_bps_max maximum bandwidth usage for inbound L2 offline business

Here is an example:

kind: ConfigMap
apiVersion: v1
metadata:
  name: terway-qos
data:
  global_bps_config: |
    hw_tx_bps_max 900000000
    hw_rx_bps_max 0
    offline_l1_tx_bps_min 100000000
    offline_l1_tx_bps_max 200000000
    offline_l2_tx_bps_min 100000000
    offline_l2_tx_bps_max 300000000
    offline_l1_rx_bps_min 0
    offline_l1_rx_bps_max 0
    offline_l2_rx_bps_min 0
    offline_l2_rx_bps_max 0

The bandwidth unit is Bytes/s, and the bandwidth limitation precision is at least 1MB or higher.

Pod bandwidth limitation configuration

Supports Kubernetes standard annotations:

  • kubernetes.io/egress-bandwidth
  • kubernetes.io/ingress-bandwidth

Supports hot update of annotations to adjust Pod bandwidth limitation.

Please note that the CNI plugin may also support Kubernetes standard annotations, which may affect the hot update. In this case, you can choose to disable the bandwidth limitation feature of the CNI plugin.

License

terway-qos developed by Alibaba Group and licensed under the Apache License (Version 2.0) This product contains various third-party components under other open source licenses. See the NOTICE file for more information.

About

The QoS project is a cloud-native solution leveraging eBPF technology, designed to efficiently manage and optimize network traffic across diverse hybrid deployment scenarios.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages