Troubleshooting

Frozen Deployments

Barcelona uses CloudFormation to handle deployments. Sometimes when deploying to staging or production it might freeze if you provide the wrong version hash e.g. bcn deploy -e staging --tag $COMMIT_HASH. The deploy will eventually timeout, but sometimes you don't want to wait around for the deployment to finish.

Solution #1

1. Login and go to services menu and select "Cloud Formation"
2. Look for the cloud formation template that says UPDATE_IN_PROGRESS
3. Select it and click "Actions -> Cancel Update"
4. Formation template will rollback to previous template and say ROLLBACK_IN_POGRESS

Note, you will need to wait several minutes for the rollback to complete before deploying again.

Solution #2 (Staging only. Do not do this on production!)

1. Login and go to services menu and select "Amazon ECS"
2. Check for the event tab of the services you are deploying, it is repeating starting task
3. Click "Update" and update the "number of tasks" to zero
4. Deployment of the ECS service will complete without succeeding to run the task
5. CloudFormation completes the update so you can deploy the fix again