Partition and Replication - Dealing with Changes

[ytyou]

Assume our cluster starts in a consistent state. All the hosts in the cluster has exactly the same partitioning configuration to start with. From here, we want to make changes to the partition. These includes,

- Same N hosts but different partitions;
- Add one or more hosts, with different partitions;
- Delete one or more hosts, with same/different partitions;
- Add and delete hosts, with same/different partitions;

Here's what each hosts do when new config is detected:

void handle_partition_config_change()
{
load:
    new_config = load_partition_config();
    if (new_config is the same as existing one)
        return;  // nothing to do

    for (each host "H" in the new_config)
    {
        if (H does not have a new config)
            sleep TIME_OUT secs;

        if (H still does not have a new config)
            goto load;

        if (H's new_config is different than mine)
        {
            sleep 10 secs;
            goto load;
        }
    }

    // now the new_config is confirmed;
    // first, collect all missing data;
    missing_data = my_data(new_config) - my_data(existing_config);
    collect_data(missing_data, existing_config);

    // now we can response to "finished collecting missing data"

    // make sure all hosts have completed collecting missing data
    for (each host "H" in the new_config)
    {
        // also make sure they are still on the agreed config
        wait_for_host_to_finish_collecting_missing_data("H");
        if (timed out) goto load;  // abort
    }

    // this is the point of no return!
    // secoond, purge data that's no longer needed
    redundant_data = my_data(existing_config) - my_data(new_config);
    purge_data(redundant_data);

    existing_config = new_config;   // update config
    return; // done
}