Endpoint metadata

[AlexanderYastrebov]

Is your feature request related to a problem? Please describe.

Load balancing algorithms (#557) require or would benefit from additional endpoint information besides currently available
scheme, ip address and port. For example slow start aka fadeIn algorithm (#1207) has to know when endpoint became available to receive traffic. Other algorithms need endpoint metrics and thus endpoint identity is required (ip address is not suitable as identity because it can be reused).

Useful endpoint metadata values are:

• identity (e.g. targetRef.uid in k8s)
• creation date (e.g. createdAt of targetRef in k8s)
• name (e.g. name of targetRef in k8s)
• weight (for weighted load balancing algorithms)

The mechanism for passing metadata should support possible future metadata extensions.

Describe the solution you would like

Endpoint metadata may be queried by skipper via infrastructure API and/or delivered to skipper proxy/algorithms via route definitions.

Within route definition the following ways are possible to pass endpoint metadata:

Marker filters

Example route:

r0: * 
-> endpointCreated("http://10.0.0.1:8080", "2020-12-18T15:30:00Z01:00") -> endpointId("http://10.0.0.1:8080", "6b6c2568-138b-11eb-835b-9f5a8a575060")
-> endpointCreated("http://10.0.0.2:8080", "2020-12-18T15:31:00Z01:00") -> endpointId("http://10.0.0.2:8080", "7daa5380-138b-11eb-b85d-7739503adcb7")
-> <roundRobin, "http://10.0.0.1:8080", "http://10.0.0.2:8080">;

• pro: readily available
• con: semantically incorrect (hacky, its not a filter after all)

Eskip backend definition

Example route:

r0: * 
-> <roundRobin, 
    "http://10.0.0.1:8080" ("createdAt=2020-12-18T15:30:00Z01:00", "id=6b6c2568-138b-11eb-835b-9f5a8a575060"), 
    "http://10.0.0.2:8080" ("createdAt=2020-12-18T15:31:00Z01:00", "id=7daa5380-138b-11eb-b85d-7739503adcb7")
>;

Syntax is subject to be discussed.

• pro: established endpoint metadata as a first class route concept
• con: requires eskip extension

Endpoint url parameters

It is possible to use url parameters because only scheme and host of endpoint url are currently used for routing.

Example route:

r0: * 
-> <roundRobin, 
    "http://10.0.0.1:8080?createdAt=2020-12-18T15:30:00Z01:00&id=6b6c2568-138b-11eb-835b-9f5a8a575060", 
    "http://10.0.0.2:8080?createdAt=2020-12-18T15:31:00Z01:00&id=7daa5380-138b-11eb-b85d-7739503adcb7"
>;

• pro: readily available
• pro: standard url parsing machinery is available
• con: hacky?

Would you like to work on it?
I would like to validate, discuss and possibly extend the ideas first.

[szuecs]

The idea to mix kubernetes IDs into backend id seem not reasonable to me.

1. it leaks data from dataclient
2. IMO IP address is enough and not explained well enough that this doesn’t make sense.
3. the IDs by kubernetes does not tell running nor ready (endpoints will be ready), how long this takes per pod is not clear.

[AlexanderYastrebov]

@szuecs

1. it leaks data from dataclient

Well, routes themselves are a leak from dataclient then :)

2. IMO IP address is enough and not explained well enough that this doesn’t make sense.

Not sure I understand, IP reuse appears to be a very real thing:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ip-reuse
spec:
  replicas: 0
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: busybox
        image: busybox
        command: ["sleep", "3600"]
        imagePullPolicy: IfNotPresent

$ kubectl scale deployment ip-reuse --replicas=10
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-6pc99   1/1     Running   0          12s   172.17.0.11   minikube   <none>           <none>
ip-reuse-5844f5c9c5-b6njc   1/1     Running   0          12s   172.17.0.7    minikube   <none>           <none>
ip-reuse-5844f5c9c5-d4g46   1/1     Running   0          12s   172.17.0.5    minikube   <none>           <none>
ip-reuse-5844f5c9c5-f5ldl   1/1     Running   0          12s   172.17.0.6    minikube   <none>           <none>
ip-reuse-5844f5c9c5-k7zn7   1/1     Running   0          12s   172.17.0.4    minikube   <none>           <none>
ip-reuse-5844f5c9c5-nh55r   1/1     Running   0          12s   172.17.0.10   minikube   <none>           <none>
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          12s   172.17.0.8    minikube   <none>           <none>
ip-reuse-5844f5c9c5-sbnhf   1/1     Running   0          12s   172.17.0.12   minikube   <none>           <none>
ip-reuse-5844f5c9c5-w5sv7   1/1     Running   0          12s   172.17.0.3    minikube   <none>           <none>
ip-reuse-5844f5c9c5-xlckj   1/1     Running   0          12s   172.17.0.9    minikube   <none>           <none>
$ kubectl scale deployment ip-reuse --replicas=1
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          30s   172.17.0.8   minikube   <none>           <none>
$ kubectl scale deployment ip-reuse --replicas=10
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-2p4lh   1/1     Running   0          21s   172.17.0.10   minikube   <none>           <none>
ip-reuse-5844f5c9c5-64pkh   1/1     Running   0          21s   172.17.0.12   minikube   <none>           <none>
ip-reuse-5844f5c9c5-fqjzh   1/1     Running   0          21s   172.17.0.9    minikube   <none>           <none>
ip-reuse-5844f5c9c5-gmsnd   1/1     Running   0          21s   172.17.0.7    minikube   <none>           <none>
ip-reuse-5844f5c9c5-k7fmm   1/1     Running   0          21s   172.17.0.11   minikube   <none>           <none>
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          54s   172.17.0.8    minikube   <none>           <none>
ip-reuse-5844f5c9c5-r996g   1/1     Running   0          21s   172.17.0.5    minikube   <none>           <none>
ip-reuse-5844f5c9c5-s7bpz   1/1     Running   0          21s   172.17.0.6    minikube   <none>           <none>
ip-reuse-5844f5c9c5-sjkmq   1/1     Running   0          21s   172.17.0.4    minikube   <none>           <none>
ip-reuse-5844f5c9c5-z9r9x   1/1     Running   0          21s   172.17.0.3    minikube   <none>           <none>

3. the IDs by kubernetes does not tell running nor ready (endpoints will be ready), how long this takes per pod is not clear.

I am not sure I can read it correctly, but what I propose are:
a) pass endpoint identity from infrastructure (IP address can not serve as an identity due to reuse)
b) pass endpoint creation time from infrastructure to reliably know when endpoint became ready to receive traffic (i.e. not guess it but trust k8s)

[AlexanderYastrebov]

In Kubernetes endpoint metadata might also include pod name which, if logged along with endpoint IP, would simplify correlation with application and event logs that contain pod name.

[szuecs]

Yes ep addresses has pod names:

subsets:
- addresses:
  - ip: 10.2.10.18
    nodeName: ip-172-31-21-23.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: <pod-name>
      namespace: <pod-namespace>