Managing overview on amazon

Getting a database shell

1. overview-manage ssh production database to log in to the database instance
2. sudo -u postgres psql overview

Creating a .psql database backup

This works even while the database is being modified. The backed-up database will have the exact same state as the database had when the backup started.

If you're backing up a database from a remote Postgres server:

1. Find the host, username, database and password of the database to back up.
2. pg_dump -v --no-owner -Fc -f backup.psql --host HOST --user USERNAME DATABASE and enter the password.

But if you're logged into the database instance, it's easier (as long as you're in a directory with the right permissions):

1. sudo -u postgres pg_dump -v --no-owner -Fc -f backup.psql

(The --no-owner is necessary when backing up from Heroku, because the Heroku username, which is gibberish, won't exist on our Amazon AWS server and so the restore will fail.)

Restoring a .psql database backup

1. Run DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL PRIVILEGES ON SCHEMA public TO overview; in a database shell.

2. Run sudo -u postgres pg_restore -v -c -d DATABASE --no-owner FILENAME. This will work, though the owner will be postgres instead of overview.

3. Run SELECT 'ALTER TABLE "' || tablename || '" OWNER TO overview;' FROM pg_tables WHERE schemaname = 'public'; in a database shell

4. Copy/paste all resulting commands back into the same database shell.

5. In a database shell, run:

    SELECT 'ALTER TYPE "' || typname || '" OWNER TO overview;' FROM (SELECT typname FROM pg_type WHERE oid IN (SELECT DISTINCT enumtypid FROM pg_enum)) x; 
   

6. Copy/paste all resulting commands back into the same database shell.

7. overview-manage restart production

Editing Overview configuration files

1. Pick ENV=(production|staging), SERVER=(worker|frontend)
2. aws s3 cp s3://overview-$ENV-secrets/$SERVER-env.sh .
3. edit $SERVER-env.sh
4. aws s3 cp $SERVER-env.sh s3://overview-$ENV-secrets/$SERVER-env.sh
5. overview-manage restart staging (or staging/web, for instance)

Editing service configuration files

1. SSH to an instance (e.g., overview-manage ssh production database) and edit the configuration file.
2. Test that your edits work.
3. Copy your edits into aws-overview-tools/script/user-data/$SERVER.txt, which follows Yaml format.

Deploying a new version

overview-manage deploy overview-server@[TAG] staging

Alternatively:

overview-manage publish overview-server@[TAG] staging overview-manage restart staging

Updating overview-manage on the manage instance

1. Push a new version to Git
2. overview-manage ssh to connect to the manage instance, then cd /opt/overview/aws-overview-tools && git pull --rebase.
3. Log out

It wouldn't be the end of the world if you modified the repository directly on the manage instance and then pushed. But there wouldn't be an author in the commit logs.

Granting or revoking access to the manage instance

Edit /home/ubuntu/.ssh/authorized_keys.

After revoking somebody's access, remember that they may still own the manage SSH private key. That shouldn't be a security concern (anything it can ssh into is firewalled), but better safe than sorry. Rotate those keys by editing /home/ubuntu/.ssh/authorized_keys on all instances, making them all use a new private key from the manage instance. Delete and create a new manage key on the EC2 Management Console, too, so future instances will use the new one.

Clearing overview-manage's caches

(Written for https://www.pivotaltracker.com/n/projects/928628/stories/69190832)

overview-manage stores files in a few places.

Let's iterate over them:

The build-cache volume

This stores Ivy and Node caches, so that the build system doesn't need to download the same files over and over again from remote repositories every time you deploy.

To wipe it, just attach the volume on the manage instance, mkfs.ext4 it, and detach it.

/opt/overview/manage

This stores bare git repositories. Just rm -rf /opt/overview/manage.