Skip to content

Commit

Permalink
documenting concurrent migrations
Browse files Browse the repository at this point in the history
  • Loading branch information
Shlomi Noach committed Jul 26, 2017
1 parent 6e51068 commit 479c228
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 4 deletions.
22 changes: 18 additions & 4 deletions doc/cheatsheet.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# Cheatsheet

### Operation modes

![operation modes](images/gh-ost-operation-modes.png)


`gh-ost` operates by connecting to potentially multiple servers, as well as imposing itself as a replica in order to streamline binary log events directly from one of those servers. There are various operation modes, which depend on your setup, configuration, and where you want to run the migration.

### a. Connect to replica, migrate on master
#### a. Connect to replica, migrate on master

This is the mode `gh-ost` expects by default. `gh-ost` will investigate the replica, crawl up to find the topology's master, and will hook onto it as well. Migration will:

Expand Down Expand Up @@ -47,7 +49,7 @@ gh-ost \
With `--execute`, migration actually copies data and flips tables. Without it this is a `noop` run.


### b. Connect to master
#### b. Connect to master

If you don't have replicas, or do not wish to use them, you are still able to operate directly on the master. `gh-ost` will do all operations directly on the master. You may still ask it to be considerate of replication lag.

Expand Down Expand Up @@ -80,7 +82,7 @@ gh-ost \
[--execute]
```

### c. Migrate/test on replica
#### c. Migrate/test on replica

This will perform a migration on the replica. `gh-ost` will briefly connect to the master but will thereafter perform all operations on the replica without modifying anything on the master.
Throughout the operation, `gh-ost` will throttle such that the replica is up to date.
Expand Down Expand Up @@ -146,7 +148,7 @@ gh-ost --allow-master-master --assume-master-host=a.specific.master.com

Topologies using _tungsten replicator_ are peculiar in that the participating servers are not actually aware they are replicating. The _tungsten replicator_ looks just like another app issuing queries on those hosts. `gh-ost` is unable to identify that a server participates in a _tungsten_ topology.

If you choose to migrate directly on master (see above), there's nothing special you need to do.
If you choose to migrate directly on master (see above), there's nothing special you need to do.

If you choose to migrate via replica, then you need to make sure Tungsten is configured with log-slave-updates parameter (note this is different from MySQL's own log-slave-updates parameter), otherwise changes will not be in the replica's binlog, causing data to be corrupted after table swap. You must also supply the identity of the master, and indicate this is a tungsten setup, as follows:

Expand All @@ -155,3 +157,15 @@ gh-ost --tungsten --assume-master-host=the.topology.master.com
```

Also note that `--switch-to-rbr` does not work for a Tungsten setup as the replication process is external, so you need to make sure `binlog_format` is set to ROW before Tungsten Replicator connects to the server and starts applying events from the master.

### Concurrent migrations

It is possible to run concurrent `gh-ost` migrations.

- Never on the exact same table.
- If running on different replicas, (e.g. `table1` on `replica1` and `table2` on `replica2`) then no further configuration required.
- If running from same server (binaries run on same server, regardless of which replica/replicas are used):
- Make sure not to specify same `-serve-socket-file` (or let `gh-ost` pick one for you).
- You may choose to use same `-throttle-flag-file` (preferably use `-throttle-additional-flag-file`, this is exactly the reason there's two, this latter file is for sharing).
- You may choose to use same `-panic-flag-file`. This all depends on your flow and how you'd like to control your migrations.
- If using same inspected box (either master or replica, `--host=everyone.uses.this.host`) then for each `gh-ost` process you must also provide a different, unique `--replica-server-id`. Optionally use process ID (`$$` in shell) ; but it's on you to choose a number that does not collide with another `gh-ost` or another running replica.
4 changes: 4 additions & 0 deletions doc/questions.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,8 @@ At this time there is no equivalent to `ALTER IGNORE`, where duplicates are impl
It is therefore unlikely that `gh-ost` will support this behavior.

### Run concurrent migrations?

Yes. TL;DR if running all on same replica/master, make sure to provide `--replica-server-id`. [Read more](cheatsheet.md#concurrent-migrations)

# Why

0 comments on commit 479c228

Please sign in to comment.