Ingress paths with multiples match does not give precedence to the longest match

[ese]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Following an upgrade from Cilium version 1.14.1 to 1.14.2, we observed a change in ingress traffic routing behaviors under specific conditions involving multiple path matches.

Issue Details:

• Ingress Configuration:

  • Service A is configured with the path /.
  • Service B has two configurations: one with the path /api and another with the regex path /(.*)/api.

• Observed Behavior:

  • In version 1.14.1, ingress traffic that matched the regex path was correctly routed to Service B, likely due to the more specific path configuration.
  • After upgrading to 1.14.2, traffic intended for the third configuration (/(.*)/api) is incorrectly routed to Service A instead of Service B.

Additionally, from version 1.14.4 to 1.14.5, a new issue emerged when TLS is enabled on ingresses. With TLS, all requests are incorrectly routed to Service A, irrespective of the specified path.

These routing errors disrupt expected network traffic flow and service delivery within our Kubernetes environment.

steps and manifests to reproduce the issue:

serviceA.yaml

apiVersion: v1
kind: Service
metadata:
  name: a-service
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: a-server
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: a-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: a-server
  template:
    metadata:
      labels:
        app: a-server
    spec:
      containers:
      - image: jmalloc/echo-server
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 8080
          name: http-port
          protocol: TCP
      restartPolicy: Always
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
  name: servicea-default
  namespace: servicea
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: a-service
            port:
              number: 80
        path: /
        pathType: Prefix

serviceB.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: b-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: b-server
  template:
    metadata:
      labels:
        app: b-server
    spec:
      containers:
      - image: jmalloc/echo-server
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 8080
          name: http-port
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: b-service
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: b-server
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
  name: b-path
  namespace: serviceb
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /api
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
  name: b-statics
  namespace: serviceb
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /exact-path
        pathType: Exact
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /(.*)/api
        pathType: Prefix

test.sh

#!/bin/bash

set -e

DOMAIN="shareddomain"
PROTOCOL="http"

curl -s ${PROTOCOL}://${DOMAIN}/ | grep a-deployment || echo "ERROR: / -> serviceA"
curl -s ${PROTOCOL}://${DOMAIN}/api/random | grep b-deployment || echo "ERROR: /api/X -> serviceB"
curl -s ${PROTOCOL}://${DOMAIN}/fr/api/random | grep b-deployment || echo "ERROR: /fr/api/X -> serviceB"
curl -s ${PROTOCOL}://${DOMAIN}/exact-path | grep b-deployment || echo "ERROR: /exact-path -> serviceB"
curl -s ${PROTOCOL}://${DOMAIN}/random | grep a-deployment || echo "ERROR: /random -> serviceA"

Ensure you change shareddomain.local for some domain reachable to your cluster

1.- Apply serviceA manifests in servicea namespace

kubectl create ns servicea
kubectl -n servicea apply -f servicea.yaml

kubectl create ns serviceb
kubectl -n serviceb apply -f serviceb.yaml

2.- Execute test.sh script for quick testing

output with 1.14.1

Request served by a-deployment-665985fc6-pngtd
Request served by b-deployment-7cd6564859-lrr8s
Request served by b-deployment-7cd6564859-lrr8s
Request served by b-deployment-7cd6564859-lrr8s
Request served by a-deployment-665985fc6-pngtd


output with cilium 1.14.2 and beyond

Request served by a-deployment-665985fc6-pngtd
Request served by b-deployment-7cd6564859-lrr8s
ERROR: /fr/api/X -> serviceB
Request served by b-deployment-7cd6564859-lrr8s
Request served by a-deployment-665985fc6-pngtd

Furthermore we have detected also a change betweent 1.14.4 to 1.14.5 where ingress is sent to the wrong service for most wide cases when you are using TLS for the ingresses.

To reproduce this just add TLS config for the same example than before and all requests will end in service A despite the path you use in the request.

Manifests for TLS:

serviceA.yaml

apiVersion: v1
kind: Service
metadata:
  name: a-service
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: a-server
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: a-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: a-server
  template:
    metadata:
      labels:
        app: a-server
    spec:
      containers:
      - image: jmalloc/echo-server
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 8080
          name: http-port
          protocol: TCP
      restartPolicy: Always
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
    cert-manager.io/cluster-issuer: letsencrypt-cluster
  name: servicea-default
  namespace: servicea
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: a-service
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
    - hosts:
      - shareddomain.local
      secretName: cert-shareddomain

serviceB.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: b-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: b-server
  template:
    metadata:
      labels:
        app: b-server
    spec:
      containers:
      - image: jmalloc/echo-server
        imagePullPolicy: Always
        name: echo-server
        ports:
        - containerPort: 8080
          name: http-port
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: b-service
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: b-server
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
  name: b-path
  namespace: serviceb
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /api
        pathType: Prefix
  tls:
    - hosts:
      - shareddomain.local
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: cilium
  name: b-statics
  namespace: serviceb
spec:
  ingressClassName: cilium
  rules:
  - host: shareddomain.local
    http:
      paths:
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /exact-path
        pathType: Exact
      - backend:
          service:
            name: b-service
            port:
              number: 80
        path: /(.*)/api
        pathType: Prefix
  tls:
    - hosts:
      - shareddomain.local

Change shareddomain.local with a domain that can reach the cluster and set PROTOCOL to https in the test.sh script

Results with TLS config:

For v1.14.1

Request served by a-deployment-665985fc6-pngtd
Request served by b-deployment-7cd6564859-lrr8s
Request served by b-deployment-7cd6564859-lrr8s
Request served by b-deployment-7cd6564859-lrr8s
Request served by a-deployment-665985fc6-pngtd

For v1.14.2, v.1.14.3, v1.14.4

Request served by a-deployment-665985fc6-pngtd
Request served by b-deployment-7cd6564859-lrr8s
ERROR: /fr/api/X -> serviceB
Request served by b-deployment-7cd6564859-lrr8s
Request served by a-deployment-665985fc6-pngtd

For v1.14.5 and beyond

Request served by a-deployment-665985fc6-pngtd
ERROR: /api/X -> serviceB
ERROR: /fr/api/X -> serviceB
ERROR: /exact-path -> serviceB
Request served by a-deployment-665985fc6-pngtd

Cilium Version

1.14.2
1.14.5

Kernel Version

6.1.84

Kubernetes Version

v1.29.1-eks-61c0bbb

Regression

1.14.1

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[sayboras]

Thanks for your issue, let me reproduce locally and get back to you soon.