Dask Setup

Create a docker image by running docker_build.sh from the infra/docker directory, and push the image to the registry by running docker_push.sh.
Use wm-playbooks/sif to create a set of VMs and configure them as a swarm.
SSH into one of the VMs and copy over docker-compose.yml file.
Deploy to docker swarm by running docker stack deploy --compose-file docker-compose.yml dask_swarm
Docker deploy will take ~15 seconds. Verify that the services are running with docker ps and check the logs with docker service logs dask_swarm_worker and docker service logs dask_swarm_scheduler
Test the dask cluster by running dask_test.py from your local machine.
To stop the docker services run docker stack rm dask_swarm

Updating the image

If not yet done, update the docker image by running docker_build.sh from the infra/docker directory, and push the image to the registry by running docker_push.sh.
SSH into the dask VM
Stop the swarm with docker stack rm dask_swarm
Pull the latest image with docker pull docker.uncharted.software/worldmodeler/wm-data-pipeline:latest
Restart docker swarm by running docker stack deploy --compose-file docker-compose.yml dask_swarm