Sharding support

[c9s]

Shard Operations

• Allocate Shard

• Remove Shard

• CloneShard (use mysqldbcopy)

  • Convert DSN into command string

    mysqldbcopy --source=user:pass@host:port:socket --destination=user:pass@host:port:socket orig_db:new_db
    
    mysqldbcopy --source=root@localhost:/opt/local/var/run/mysql56/mysqld.sock \
                --destination=root@localhost:/opt/local/var/run/mysql56/mysqld.sock \
                shade_src:shade_dst
    

• Prune Shard

  • Iterate schemas
  • Prune collection with the sharding key.
    • Get the shard key
    • Query distinct values from collection.
    • Calculate shard Id on each value.
    • For each value, remove the rows that don't belong to the shard.

• Move Chunk

  • Provide import method in Repository

    $repo->import($book);
    
    When importing a record into different repo, we should remove the auto-increment key.
    
    $repo->duplicate($book);
    
    When duplicating a record, all the details of the record should be the same in the new repository.
    

  • Provide moveTo method in BaseModel

    $book->moveTo($repo2);
    

  • Move a shard to another instance.

• Reshard

• Split Chunk

  • Implement a consistent hash
  • Add target by pre-computed index.
  • Implement ChunkManager to initialize a chunk list with specific range.

Aggregation Query

• BroadcastQuery
  • Broadcast SQL statement to all shards.
• Extract SQL building method for create (insertion)
• Extract SQL building method for update
• Extract SQL building method for delete
• Map SQL and arguments to QueryWorker and then reduce the result...
• QueryDispatcher

  • Given a query, return Repo objects with different connections and
    run queries on these nodes.

  • Shard by Range

  • Virtual Shards (by using larger key space)

Instance Management Commands

• instance add
• instance remove