Loadbalancer health checks report false positives

[rochecompaan]

🐛 Describe the bug

AWS load balancer health checks in the full stack template false reports that a node is available because a TCP connection can be established on port 6443, even if an HTTP request to the port hangs.

Since one cannot currently make anonymous HTTP requests to the Kubernetes API port, I recommend that we switch to an HTTP health check on a backend service like Django. We can point the health check to the same view as the Django readiness check.

💭 Expected Behavior

Load balancer health checks should ensure that traffic is only routed to nodes where nodes are healthy.

🔁 Steps To Reproduce

It's difficult to reproduce a kubelet process hanging. One could add a dummy service that never responds on a different port and update the existing TCP health check to connect to that port instead.

🌎 Environments

• Sandbox
• Staging
• Production

📝 Additional context

No response

📜 Relevant log output

📸 Screenshots

No response