Topology Aware Routing not working

[roman5595]

What happened?

Despite enabling topology-aware routing, I'm still seeing traffic being routed from one AZ to another. This is not the expected behavior, as it should prioritize in-zone traffic.

Current setup:

3 AZ eu-west
3 pods in each AZ
3 endpoints per each AZ

What did you expect to happen?

When i enable topology aware routing I expect that traffic will stay in AZ where traffic was generated to maximize cost savings.

How can we reproduce it (as minimally and precisely as possible)?

1. create nginx deployment with 9 replicas with :
   spec:
   topologySpreadConstraints:
   - maxSkew: 1
   topologyKey: topology.kubernetes.io/zone
   whenUnsatisfiable: DoNotSchedule
   labelSelector:
   matchLabels:
   app: nginx
2. create Service
   apiVersion: v1
   kind: Service
   metadata:
   annotations:
   service.kubernetes.io/topology-aware-hints: Auto
   name: nginx
   spec:
   ports:

• port: 80
  protocol: TCP
  targetPort: 80
  selector:
  app: nginx
  type: ClusterIP

3. check for endpointslice if hints were generated
4. Label every node with : topology.kubernetes.io/zone

Anything else we need to know?

kubectl get events --field-selector involvedObject.kind=Service,involvedObject.name=nginx
16m Normal TopologyAwareHintsEnabled service/nginx Topology Aware Hints has been enabled, addressType: IPv4

Kube-Proxy logs:

I0505 16:46:38.759957 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:46:38.105331 10 topology.go:171] "Skipping topology aware endpoint filtering since one or more endpoints is missing a zone hint"
I0505 16:49:27.896972 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"
I0505 16:49:28.432488 10 topology.go:181] "Skipping topology aware endpoint filtering since no hints were provided for zone" zone="eu-west-1c"

Hints were assigned:
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:

• addresses:
  • 100.96.11.100
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1c
      nodeName: i-06c703e12f8392a11
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-jdnd9
      namespace: default
      uid: a142d4d9-e078-44e9-9ce1-3956faa1e5ce
      zone: eu-west-1c
• addresses:
  • 100.96.11.155
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1c
      nodeName: i-06c703e12f8392a11
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-cr7f2
      namespace: default
      uid: a2bd185c-86d0-47a8-a5ab-45ff4e32a85d
      zone: eu-west-1c
• addresses:
  • 100.96.9.173
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1b
      nodeName: i-088e764bd881db13b
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-znbbm
      namespace: default
      uid: 204eaf01-79bb-4993-a37c-b7bb90fb2094
      zone: eu-west-1b
• addresses:
  • 100.96.9.26
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1b
      nodeName: i-088e764bd881db13b
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-lfxqp
      namespace: default
      uid: 47909cec-d536-4bee-ac95-5b021a29041b
      zone: eu-west-1b
• addresses:
  • 100.96.10.164
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1a
      nodeName: i-05ab65b1b0ba1ab03
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-mjvd5
      namespace: default
      uid: 777dceea-a87f-483d-ac7f-d30cd7812bb1
      zone: eu-west-1a
• addresses:
  • 100.96.10.225
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1a
      nodeName: i-05ab65b1b0ba1ab03
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-ltj2v
      namespace: default
      uid: 2a4d680c-8822-4a2c-b3b8-a56ceaff96e7
      zone: eu-west-1a
• addresses:
  • 100.96.11.62
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1c
      nodeName: i-06c703e12f8392a11
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-xwqlm
      namespace: default
      uid: b4fb49ed-630b-4f7c-8ea0-22cc11ab2c9d
      zone: eu-west-1c
• addresses:
  • 100.96.9.138
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1b
      nodeName: i-088e764bd881db13b
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-p92c5
      namespace: default
      uid: 48c29d22-98f4-4316-bc96-958d608a8ffc
      zone: eu-west-1b
• addresses:
  • 100.96.12.233
    conditions:
    ready: true
    serving: true
    terminating: false
    hints:
    forZones:
    • name: eu-west-1a
      nodeName: i-0272547448c4bb6b7
      targetRef:
      kind: Pod
      name: nginx-84bdbbf4bb-sqb79
      namespace: default
      uid: 3f206f30-184c-4c83-986f-29b7b4ad5f77
      zone: eu-west-1a
      kind: EndpointSlice

Kubernetes version

1.26.12

Cloud provider

AWS

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vaibhav2107]

/sig network

[sftim]

Topology Aware Routing adjusts routing behavior to prefer keeping traffic in the zone it originated from. In some cases this can help reduce costs or improve network performance.

(my emphasis)
I think the docs are correct, and that the implementation behaves as documented.

If you'd like to enforce that traffic stays inside a particular zone, that would be a feature request @roman5595.
/priority awaiting-more-evidence

---

Kubernetes is open source; if you'd like to improve the topology aware routing heuristics, we welcome code contributions.

[thockin]

@robscott