Improve our system of paired deployments / operations

[choldgraf]

Context

We recently had an incident that occurred because of a mistake made while decommissioning some cloud infrastructure, reported in:

• [Incident] CarbonPlan AWS hub had running infrastructure we didn't track #1666

The error that we made was that we incorrectly decommissioned the cluster, and didn't double-check that it was entirely shut down. As a result it started accruing cloud costs in the background. Because these cloud costs weren't too high, it went unnoticed for some time.

We should expect that our team will make mistakes like this - it is normal human nature. To reduce the risk associated with it, we should have a system of team checks that make us more likely to catch these kinds of issues in the future.

Proposal

I propose that we implement a system of paired deployments whenever we perform an operation in the cloud infrastructure. The goal of paired deployments is to:

• Provide at least two pairs of eyes to double-check work
• Provide assistance and support when debugging and changing infrastructure
• Provide an opportunity to learn and share knowledge among the team

This could be done either synchronously (by having live paired deployment sessions) or asynchronously (by having two team members assigned on an issue, and asking each of them to confirm that it has been completed as expected).

Updates and actions

No response

[yuvipanda]

Handled by various other improvements in our processes.