|
1 | | -# Limitations |
| 1 | +# Lightpath - a kube-proxy replacement based on Envoy |
| 2 | + |
| 3 | +Lightpath consists of multiple data plane and control plane components: |
| 4 | +1. **Control Plane**: |
| 5 | + - `controlplane` (node-scoped): runs on every node and acts as control server for Envoy |
| 6 | + - `webook` (cluster-scoped): The webhook components intercepts the service creation and updates the handling proxy to Lightpath. This allows a hybrid deployment next to the kube-proxy |
| 7 | +2. **Data Plane**: |
| 8 | + - `proxy`(node-scoped): Envoy acts as proxy instance and is configured to connect to the node-local control plane |
| 9 | + - `redirect` configures the iptables redirect targets for ClusterIPs handled by Lightpath |
2 | 10 |
|
3 | | -- Single IP address per endpoint (no pods with multiple IPs supported) |
4 | | -- as a PoC, the TCP protocol is targeted |
| 11 | +# Limitations |
| 12 | +As a PoC, the TCP and HTTP protocol are targeted. STCP and UDP based protocols are not supported (yet) and need to be handled by kube-proxy. Therefore, a hybrid setup is required for full service connectivity. |
5 | 13 |
|
6 | 14 | # Deployment |
7 | | -> Injection is opt-out right now for the label `lightpath.cloud/proxy: disabled`. Will be changed to opt-in later on |
8 | 15 |
|
9 | | -lightpath has two components: |
10 | | -- a mutating webhook, which injects the `service.kubernetes.io/service-proxy-name=lightpath` label (when not existing), creating an "opt-out" enabling of the lightpath proxy |
11 | | -- a proxy component running on each node, managing IPTable redirects for its services (will potentially be replaced by eBPF) |
| 16 | +Lightpath can be deployed on any Kubernetes Cluster with an iptables based CNI (e.g. Calico, Flannel, ...). |
| 17 | + |
| 18 | +## Test cluster |
| 19 | +The `make k8s-up` command bootstraps a new [Kubernetes in Docker](https://kind.sigs.k8s.io/docs/user/quick-start/), which can be used for further testing |
| 20 | + |
| 21 | +## Deploying |
| 22 | +> Lightpath images are (publicly) available for linux/amd64 and linux/arm64. |
| 23 | +
|
| 24 | +The deployment manifests are based on [Kustomize](http://kustomize.io) and are generated with `kustomize build deploy/default/`. Lightpath requires cert-manager to be installed on the tareted cluster; the installation will fail otherwise. The `make k8s-up` command already setups all required dependencies. |
| 25 | + |
| 26 | +As a shortcut, `make deploy` applies the kustomize manifests to Kubernetes results. |
| 27 | + |
| 28 | +A test deployment running a simple echo server can be applied to a cluster with `kubectl apply -f hack/ci/test-deploy.yaml`. |
| 29 | +A new pod with an interactive shell can be bootstrapped using e.g. `kubectl run tmp-net-debug-shell -it --rm --image nicolaka/netshoot -- /bin/bash`, which can then be used to access the deployed service: |
| 30 | +``` |
| 31 | +bash-5.1# curl whoami.whoami.svc |
| 32 | +Hostname: whoami-5978d4b87d-dwx2t |
| 33 | +IP: 127.0.0.1 |
| 34 | +IP: ::1 |
| 35 | +IP: 10.244.2.3 |
| 36 | +IP: fe80::e8f1:57ff:fefa:2e66 |
| 37 | +RemoteAddr: 172.18.0.3:42850 |
| 38 | +GET / HTTP/1.1 |
| 39 | +Host: whoami.whoami.svc |
| 40 | +User-Agent: curl/7.83.1 |
| 41 | +Accept: */* |
| 42 | +X-Envoy-Attempt-Count: 1 |
| 43 | +X-Envoy-Expected-Rq-Timeout-Ms: 5000 |
| 44 | +X-Forwarded-Proto: http |
| 45 | +X-Request-Id: c686bc0b-1e33-445c-b945-9ae883313303 |
| 46 | +``` |
| 47 | + |
| 48 | +The X-Envoy headers indicate that the request has been successfully handled by lightpath. Furthermore, the Envoy admin interface can ben accessed by e.g. port-forwarding the configured admin port (15000) to the local machine: |
| 49 | +```bash |
| 50 | +$ kubectl port-forward $(kubectl get pod -oname -l app.kubernetes.io/name=proxy | head -n 1) 15000:15000 & |
| 51 | + |
| 52 | +$ # E.g. retrieve the configuration dump: |
| 53 | +$ curl http://localhost:15000/config_dump |
| 54 | +# ... |
| 55 | +``` |
| 56 | + |
| 57 | +# Repository Structure |
| 58 | + |
| 59 | +Overall, the repository acts as mono-repository for all lightpath components, configuration and performed benchmarks performed throughout the master thesis. |
| 60 | + |
| 61 | +The actual application source code is distributed in the `cmd`, `pkg` and `internal` folders, with the `cmd/{controlplane,redirect}` holding the two Golang based services. |
| 62 | +The webhook is implemented in the Rego language as part of the [open policy agent](https://www.openpolicyagent.org) in the Kustomize based Kubernetes manifests (`deploy/{proxy,redirect,webhook,controlplane}`). The `deploy/default` Kustomization acts as meta package for all components. |
| 63 | + |
| 64 | +The main Golang packages are located in `pkg/state/` and `pkg/translations/`. The translation package defines the actual mapping between Kubernetes resources and the Envoy representation. |
| 65 | + |
| 66 | +Performed benchmark/load tests and scalability analysis is included in the `hack/benchmark` folder, defining the infrastructure in `hack/benchmark/bench-infra` distributed across a monitoring cluster and workload cluster. The actual experiments performed are located alongside the [Fortio](https://fortio.org) results in `hack/benchmark/experiments/{scenarios,results}`. |
| 67 | +All experiments base on the `hack/benchmark/experiments/generator` environment, which has been used to generate the scenarios. |
| 68 | + |
| 69 | +The `bpf` directory includes early steps towards eBPF based forwarding of traffic to the node-local proxy and has not been covered in the thesis and is in a non functional state, but shipped for completeness. |
0 commit comments