Deploying k8s to Azure does not automatically obey Network Policies

[sgibson91]

As we have discovered in jupyterhub/mybinder.org-deploy#1154 (comment), network policies defined in helm charts won't automatically be obeyed as expected when using a k8s cluster on Azure.

To enforce this, the k8s cluster needs to be deployed into a vnet with network policies enabled. Currently, this must be done at the deployment stage and cannot be retroactively applied to existing clusters.

I can contribute some documentation on how to set up a vnet with network policies and deploy the cluster into it 🙂

[consideRatio]

@sgibson91 yepp!

Kubernetes does not by default come with something, called a controller, that enforces what is described in k8s networkpolicy resources. On GKE you can get that automatically installed if you press something similar in the k8s cluster setup.

For the documentation, it should be a general note about this alongside specific cloud provider implementation details on how to go about setting up a controller that can enforce the policies.

In the CI/CD testing of this repo, we install this manually into the k8s cluster - project calico.

zero-to-jupyterhub-k8s/dev

Lines 136 to 162 in de3a30b

	# To test network policies, we need a custom CNI like Calico. We have disabled
	# the default CNI through kind-config.yaml and will need to manually install a
	# CNI for the nodes to become Ready.
	print("Installing a custom CNI: Calico (async, in cluster)")
	_run(
	cmd=[
	"kubectl", "apply",
	"-f", "https://docs.projectcalico.org/v3.10/manifests/calico.yaml",
	],
	print_end="",
	)
	# NOTE: daemonset/calico-node pods' main container fails to start up without
	# an additional environment variable configured to disable a check
	# that we fail.
	#
	# env:
	# - name: FELIX_IGNORELOOSERPF
	# value: "true"
	_run(
	cmd=[
	"kubectl", "patch", "daemonset/calico-node",
	"--namespace", "kube-system",
	"--type", "json",
	"--patch", '[{"op":"add", "path":"/spec/template/spec/containers/0/env/-", "value":{"name":"FELIX_IGNORELOOSERPF", "value":"true"}}]',
	],
	)