bg-kube

An interface for automating blue-green deployments on a Kubernetes cluster.

Please note that this project is in initial development and it might not be production-ready yet. Use with caution.

Features

• Publish/Rollback functions.
• Dynamic variables in YAML configuration files.
• Smoke tests for health checking before promoting a new environment.
• Easily extensible to support multiple cloud providers (other than just GKE and AWS/kops).
• Minimal setup/resources - does not live in the cloud and can be invoked from a CI service like Travis.

Workflow

1. Builds and tags a container image from a Dockerfile using docker build command.
2. Pushes the tagged image to the container registry (GCR and ECR only at this point).
3. Creates a Job workload for the database migrations (Optional - should be backwards compatible).
4. Creates a Deployment workload using the new image.

5. Creates a Service for health checking which runs the specified smoke tests command (Optional).

   • If the tests were successful, it updates the public Service workload to point to the new deployment.
   • If the tests have failed, the public service remains unaffected.

See it in action

Check out this blog post.

Installation

$ pip install bg-kube

Prerequisites

• Docker
• Kubernetes command-line tool

Google Kubernetes Engine

• Create project
• Create cluster
• Install SDK
• Login using gcloud init
• Select project using gcloud config set project <project-id>

AWS using kops

• Install AWS CLI
• Install kops CLI
• Setup environment

Minimal configurations example

Service Config - Public

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: $DOMAIN_NAME.
  labels:
    run: $SERVICE_RUN_LABEL
  name: $SERVICE_NAME
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: $SERVICE_PORT
    targetPort: $CONTAINER_PORT
  selector:
    run: $SERVICE_RUN_LABEL
    color: $COLOR
    type: pod
  type: LoadBalancer

Service Config - Health Checks

apiVersion: v1
kind: Service
metadata:
  labels:
    run: $SERVICE_RUN_LABEL
  name: $SMOKE_SERVICE_NAME
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: $SERVICE_PORT
    targetPort: $CONTAINER_PORT
  selector:
    run: $SERVICE_RUN_LABEL
    color: $COLOR
    type: pod
  type: LoadBalancer

Deployment Config

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    color: $COLOR
    run: $SERVICE_RUN_LABEL
  name: $DEPLOYMENT_NAME-$COLOR
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      color: $COLOR
      run: $SERVICE_RUN_LABEL
  template:
    metadata:
      labels:
        run: $SERVICE_RUN_LABEL
        color: $COLOR
        tag: "$TAG"
        type: pod
    spec:
      containers:
      - command: ["gunicorn", "django_app.wsgi", "--name", "todoapp", "-b", ":$CONTAINER_PORT"]
        env:
        - name: ENV
          value: $ENV
        - name: DB_URL
          value: $DB_URL
        image: $IMAGE_NAME:$TAG
        name: $CONTAINER_NAME
        ports:
        - containerPort: $CONTAINER_PORT
          protocol: TCP

env

ENV=prod
DB_URL=postgres://user:[email protected]:5432/todoapp

IMAGE_NAME=gcr.io/todoapp-12345/todo-api
CONTAINER_PORT=8000
CONTAINER_NAME=cnt-todo-api

PROJECT_NAME=todoapp-12345
CLUSTER_NAME=todoapp-cluster
CLUSTER_ZONE=us-central1-a
DOMAIN_NAME=todoapp.example.com

SERVICE_PORT=80
SERVICE_NAME=svc-todo-api
SERVICE_CONFIG=./config/service.yaml
SERVICE_RUN_LABEL=todo-api

DEPLOYMENT_NAME=dep-todo-api
DEPLOYMENT_CONFIG=./config/deployment.yaml

Publish using

$ bg-kube --env-file .env.prod publish

Arguments

positional arguments:
  {publish,rollback,build,push}
  command_args

optional arguments:
  -h, --help: show this help message and exit
  -e ENV_FILE, --env-file ENV_FILE: .env file for the options below and application vars in the configs
  -c CLUSTER_NAME, --cluster-name CLUSTER_NAME: unique name of the cluster
  -z CLUSTER_ZONE, --cluster-zone CLUSTER_ZONE: zone name of the cluster location
  -m DOCKER_MACHINE_NAME, --docker-machine-name DOCKER_MACHINE_NAME: name of the docker machine if applicable
  --docker-build-args DOCKER_BUILD_ARGS arguments supplied to docker build command separated with spaces
  -i IMAGE_NAME, --image-name IMAGE_NAME: name of the container image to build using docker
  -s SERVICE_NAME, --service-name SERVICE_NAME: name of the main service intended to serve clients
  --service-config SERVICE_CONFIG: config of the main service
  --service-timeout SERVICE_TIMEOUT timeout secs to wait for healthy state or return an error
  --deployment-config DEPLOYMENT_CONFIG config of the deployment containing the main service pods
  --deployment-timeout DEPLOYMENT_TIMEOUT timeout secs to wait for healthy state or return an error
  -x CONTEXT, --context CONTEXT: docker context path used to build the container image
  -d DOCKERFILE, --dockerfile DOCKERFILE: Dockerfile path
  --smoke-service-config SMOKE_SERVICE_CONFIG config of the smoke service lb exposed for health checks
  --smoke-tests-command SMOKE_TESTS_COMMAND: shell command to run health checks against the smoke service
  --db-migrations-job-config-seed DB_MIGRATIONS_JOB_CONFIG_SEED: job config to populate the database with initial data
  --db-migrations-job-timeout DB_MIGRATIONS_JOB_TIMEOUT timeout secs to wait for healthy state or return an error
  --db-migrations-status-command DB_MIGRATIONS_STATUS_COMMAND: shell command executed on any of the running deployment pods to return the current migrations status
  --db-migrations-apply-command DB_MIGRATIONS_APPLY_COMMAND: shell command executed on any of the running deployment pods to apply the latest migrations generated in the current image
  --db-migrations-rollback-command DB_MIGRATIONS_ROLLBACK_COMMAND: shell command executed on any of the running deployment pods with the migrations status command stdout as argument -   retrieved before applying migrations, to perform a rollback to that state
  --kops-state-store KOPS_STATE_STORE: aws cluster state storage bucket name
  --container-registry CONTAINER_REGISTRY: container registry alias or implementation class

Future Improvements

• Checks to enforce database migrations are backwards compatible
• Support for more cloud providers
• Better test coverage