Automate EC2 backups to S3

[waldoj]

• entire server
• entire application directory
• each Docker container
• ckanapi-generated site backups

[waldoj]

We might be able to add redundancy by storing the data on S3 (per #6).

[waldoj]

Seems to me like a useful backup function would be running ckanapi full-site backups every 24 hours, and saving them on S3. I'm not sure how one would go about scripting that centrally, but perhaps that's a cron job that needs to be included in each Docker instance.

[waldoj]

I suppose I've got two goals here. The first is internal redundancy, so that's easy to get back up and running if something blows up on the server. The second is client redundancy, so that if I'm hit by a bus at the same time that the AWS hosting facility is hit by a meteor, clients have a method of getting their data.

[mafintosh]

AWS has a way to snapshot an entire instance into an AMI that can be stored on S3. This requires halting the instance for a few seconds though.

[waldoj]

I've been using ec2-automate-backup, which seems to work pretty well, but I suspect there are more moderns methods of accomplishing this. :)

[waldoj]

BTW, @mafintosh @maxogden and @Karissa, I have no idea why this repo has all of y'all following it by default. GitHub permissions confuse me. Obviously, you're welcome to do so, but please don't feel any obligation—it's in no way intentional!

[waldoj]

Note that the site data is stored in /home/ubuntu/.datacats/multisite/sites, which is weird but OK.