To deploy new data pipeline code in the AWS production environment, follow these steps:
- A Docker image with the
latest
tag is automatically created during CI when changes are merged into themaster
branch. - Alternatively, manually build and push the image using the
./build_and_push.sh
script.
-
Update the Dask scheduler and workers running in the AWS Kubernetes cluster with the newer Docker image.
-
If you use the
latest
Docker tag for the pods, recreate the pods for the Dask cluster:First load the kube configuration file for the target environment
export KUBECONFIG=~/.kube/causemos-staging.config
or
export KUBECONFIG=~/.kube/causemos-prod.config
Delete the pods for the dask cluster
kubectl get po -n causemos | grep dask kubectl delete po POD -n causemos
and verify the image hash before/after
kubectl describe po POD -n causemos | grep sha
- To register new flow code, run the
register_flows.sh
script. - For test flows, run
register_test_flows.sh
. - Run the script with
-p
option flag for the production deployment e.g.register_flows.sh -p