Kubernetes 1.22 api removed beta ingress

[szuecs]

IngressesClusterURI for example is using beta and our tests should also be changed to return v1 networking objects.

Related #1477 #1478

according to https://www.civo.com/learn/migrating-your-ingresses-in-k3s-1-20 spec change is also in service that now has

port:
  number:
  name:

see also https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

[AlexanderYastrebov]

Documentation examples should be updated as well

[tomekit]

I am updating to Kubernetes v1.22 and after convincing the ingress-controller to cooperate: zalando-incubator/kube-ingress-aws-controller#448

I am now trying to run skipper, so far I've modified apiVersion from rbac.authorization.k8s.io/v1beta1 to rbac.authorization.k8s.io/v1 in ClusterRole and ClusterRoleBinding.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: skipper-ingress
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: skipper-ingress
rules:
  - apiGroups: ["extensions"]
    resources: ["ingresses", ]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["namespaces", "services", "endpoints", "pods"]
    verbs: ["get", "list"]
  - apiGroups:
      - zalando.org
    resources:
      - routegroups
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: skipper-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: skipper-ingress
subjects:
  - kind: ServiceAccount
    name: skipper-ingress
    namespace: kube-system

Unfortunately I am getting the:

[APP]time="2021-10-19T22:36:49Z" level=error msg="error while receiveing initial data;failed to load cluster state: resource not found"
[APP]time="2021-10-19T22:36:52Z" level=error msg="error while receiveing initial data;failed to load cluster state: resource not found"
10.107.1.143 - - [19/Oct/2021:22:36:53 +0000] "GET /kube-system/healthz HTTP/1.1" 404 10 "-" "ELB-HealthChecker/2.0" 0 10.107.50.32:9999 - -

Any idea why this might be?

Skipper.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: skipper-ingress
  namespace: kube-system
  labels:
    component: ingress
spec:
  selector:
    matchLabels:
      component: ingress
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: skipper-ingress
      labels:
        component: ingress
        application: skipper
    spec:
      hostNetwork: true
      serviceAccountName: skipper-ingress
      containers:
        - name: skipper-ingress
          image: registry.opensource.zalan.do/pathfinder/skipper:latest
          ports:
            - name: ingress-port
              containerPort: 9999
              hostPort: 9999
            - name: metrics-port
              containerPort: 9911
          args:
            - "skipper"
            - "-kubernetes"
            - "-kubernetes-in-cluster"
            - "-address=:9999"
            - "-proxy-preserve-host"
            - "-serve-host-metrics"
            - "-enable-ratelimits"
            - "-experimental-upgrade"
            - "-metrics-exp-decay-sample"
            - "-lb-healthcheck-interval=3s"
            - "-metrics-flavour=codahale,prometheus"
            - "-enable-connection-metrics"
            - "-response-header-timeout-backend=3605s"
            - "-write-timeout-server=3605s"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          readinessProbe:
            httpGet:
              path: /kube-system/healthz
              port: 9999
            initialDelaySeconds: 5
            timeoutSeconds: 5

I smell something related to RBAC, but don't have enough experience to nail it down quickly.

[mikkeloscar]

For RBAC you need to replace this part:

  - apiGroups: ["extensions"]
    resources: ["ingresses", ]
    verbs: ["get", "list"]

with:

  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", ]
    verbs: ["get", "list"]

[tomekit]

Thanks @mikkeloscar.
After amending extensions to networking.k8s.io I am now getting slightly different error:

[APP]time="2021-10-20T08:05:34Z" level=error msg="error while receiveing initial data;failed to load cluster state: request failed, status: 403, 403 Forbidden"

currently it responds with 403, whilst previously it was resource not found:

[APP]time="2021-10-19T22:24:50Z" level=error msg="error while receiveing initial data;failed to load cluster state: resource not found"

[mikkeloscar]

Ok, then the issue is skipper is looking for the old api extensions and now you removed permissions for that so it's 403 instead of 404.

So the real solution includes changes to skipper to use the newer API, similar to what was done for kube-ingress-aws-controller.

[szuecs]

Yes we need to change the API path, but also change the service port, because spec changed there.

[tomekit]

Yes an example of previous extensions/v1beta1 backend:

 paths:
  - path: /
    backend:
      serviceName: app-web
      servicePort: 80

and networking.k8s.io/v1 :

paths:
  - path: /
    pathType: Prefix
    backend:
      service:
        name: app-web
        port:
          number: 80

Is it something you will have chance working anytime soon? Can you give any non-binding ETA?

I am happy at least to test those changes on my cluster.

[szuecs]

@tomekit right now I can't give an ETA, because I have no time, but next week one colleague is back and maybe it changes soon.

[szuecs]

@tomekit I implemented kubernetes ingress V1 and you can test report if it works great.
You can find a test image at registry.opensource.zalan.do/teapot/skipper-test:pr-1892-4.
You have to enable it with -kubernetes-ingress-v1