ImageSwap Mutating Admission Controller for Kubernetes

This Admission Controller will swap an existing image definition in a Pod with a user specified image prefix. This allows you to use the same manifests for airgapped environments that don't have access to commonly used image registries (dockerhub, quay, gcr, etc.).

The webhook is written in Python using the Flask framework.

Example

Existing Image Definition:

docker.io/nginx/nginx:latest

Image After the Swap:

my-registry.example.com/nginx/nginx:latest

Overview

• Prereqs
• Quickstart
• Health Check
• Image
• Configuration
• Advance Install
• Metrics
• Testing
• Cautions
• Troubleshooting
• Contributing

Prereqs

Kubernetes 1.9.0 or above with the admissionregistration.k8s.io/v1beta1 (or higher) API enabled. Verify that by the following command:

$ kubectl api-versions | grep admissionregistration.k8s.io/v1beta1

The result should be:

admissionregistration.k8s.io/v1beta1

In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.

Permissions

ImageSwap requires cluster-admin permissions to deploy to Kubernetes since it requires access to create/read/update/delete cluster scoped resources (MutatingWebhookConfigurations, Certificates, etc.)

Quickstart

You can use the following command to install ImageSwap from this repo with sane defaults

NOTE: The quickstart installation is not meant for production use. Please read through the Cautions sections, and as always, use your best judgement when configuring ImageSwap for production scenarios.

$ kubectl apply -f https://raw.githubusercontent.com/phenixblue/imageswap-webhook/master/deploy/install.yaml

This will do the following

• Create the imageswap-system namespace
• Create cluster and namespace scoped roles/rolebindings
• Deploy the ImageSwap workload and related configs

Once this is complete you can do the following to test

Create and label a test namespace

$ kubectl create ns test1
$ kubectl label ns test1 k8s.twr.io/imageswap=enabled

Deploy some test workloads

# These examples assume you're in the root directory of this repo
# Example with without expected prefix

$ kubectl apply -f ./testing/deployments/test-deploy01.yaml -n test1

# Example with expected prefix

$ kubectl apply -f ./testing/deployments/test-deploy02.yaml -n test1

Image

ImageSwap uses a couple of images for operation

• imageswap-init
• imageswap

Init Container

ImageSwap uses the imageswap-init init-container to generate/rotate a TLS cert/key pair to secure communication between the Kubernetes API and the webhook. This action takes place on Pod startup.

Configuration

Change the IMAGE_PREFIX environment variable definition in the imageswap-env-cm.yaml manifest to customize the repo/registry for the image prefix mutation.

Metrics

Prometheus formatted metrics for API rquests are exposed on the /metrics endpoint.

Testing

Assuming you've followed the quickstart steps

• Review Deployment and Pod spec to validate the webhook is working

  $ kubectl get deploy hello-world -n test1 -o yaml
  $ kubectl get pods -n test1
  $ kubectl get pod <pod_name> -n test1 -o yaml

  NOTE: You should see the swapped image definition instead of the original definition in the test-deploy.yaml manifest.

Cautions

Production Considerations

• By Default the ImageSwap Mutating Webhook Configuration is set to fail "closed". Meaning if the webhook is unreachable or doesn't return an expected response, requests to the Kubernetes API will be blocked. Please adjust the configuration if this is not something that fits your environment.
• ImageSwap supports operation with multiple replicas that can increase availability and performance for critical clusters.
• The certificate generated by the imageswap-init container is valid for 12 months and will be automatically rotated once the Pod restarts within 6 months of expiration. If the certificate expires, calls to the webhook wil fail. Make sure you plan for this certificate rotation.

Break Glass Scenarios

ImageSwap can be enabled and disabled on a per namespace basis by utilizing the k8s.twr.io/imageswap label on namespace resources. In emergency situations the label can be removed from a namespace to disable image swapping in that namespace.

If there are cluster-wide issues you can disable ImageSwap completely by removing the imagewap-webhook Mutating Webhook Configuration and deleting the ImageSwap deployment.

Troubleshooting

Run Docker Image Locally

$ docker run -p 5000:5000/tcp -it imageswapwebhook_app bash
$ ./deny-env.py

Access Kubernetes Service without Ingress/LB

$ kubectl get pods # to get the name of the running pod
$ kubectl port-forward <pod_name> 5000:5000

Use Curl to perform HTTP POST to webhook server

$ curl -vX POST https://localhost:5000/ -d @test.json -H "Content-Type: application/json"

Follow logs of the webhook pod

$ kubectl get pods # to get the name of the running pod
$ kubectl logs <pod_name> -f