Vitess v2.1.0-rc.1

This is the first release candidate of the 2.1 release.

We expect minimal changes between this and the final 2.1 release.

Download

• Server binaries are available in the Docker image "vitess/lite:v2.1.0-rc.1".
• Java client packages are published on Maven Central.
• Source code is available via the tag "v2.1.0-rc.1".

Upgrading

All Vitess 2.1 changes are backward compatible with the 2.0 release. Please make sure that you follow our recommended upgrade order.

Changes since the 2.0 Release

• Our website vitess.io has a new logo! :)
• Kubernetes:
  • Added Helm chart and support for StatefulSet.
• all:
  • Topology:
    • Rewrote plugins for etcd and Zookeeper (etcd2, zk2). Please see the doc how to migrate to the new storage format.
      Command line flags have changed as well (shipped examples are updated).
    • New plugin: Consul.
  • Internal: We have changed the internal error representation in the RPC messages in a backward compatible way.
• clients:
  • Java:
    • TLS support in client and JDBC driver.
    • Unified package structure under new root io.vitess.
    • 2.1.0-rc.1 packages are published on Maven Central e.g. our JDBC driver as artifact vitess-jdbc.
  • Map Reduce:
    • Faster execution on tables with a non-uniform distribution by sampling rows and splitting the table into equal-sized row ranges.
• vtctld:
  • New Angular 2 based UI.
  • Online Schema Swap: Apply complex schema changes without any downtime.
  • Resharding workflow. Automatically runs the resharding steps and shows them in the Workflow UI Tab. New tutorials added which use the workflow.
• vtgate:
  • 2PC: Support for distributed transactions.
  • Native MySQL server protocol (alpha).
  • Direct access to shards via "keyspace:shard" database name.
  • Buffer master traffic during failovers and retry to avoid client errors (disabled by default).
  • Support for SHOW statements to expose Vitess metadata.
  • Increase large installation scalability with (optional) two-layer vtgate pools (l2vtgate, applicable to 100+ shard installations).
• vttablet:
  • Native MySQL client protocol (instead of using libmysqlclient) (disabled by default).
  • Support for MySQL Row-based-Replication e.g. needed when resharding (alpha).
  • Update Stream: Applications can subscribe to a change stream (e.g. to invalidate an app level cache).
  • Messages: Native support for a queue processing framework (alpha).
  • Hot Row Protection: Serializes write transaction for the same row (range) to prevent that all transaction pool slots can be exhausted (disabled by default).
  • Transaction Throttler: Rejects new transactions with retryable error if replication lag on replicas is too high (disabled by default).

Other Noteworthy Changes

• Minimum required Go version changed to 1.8 (from 1.5).
• Updated to gRPC 1.0.4 (from 1.0.0).

Vitess v2.1.0-alpha.1

This is the first alpha release of the upcoming 2.1 version.

2.1 will be a feature release which will bring the following new features:

• Support for distributed transactions, using 2 phase commit.
• Resharding workflow improvements, to increase manageability of the process.
• Online schema swap, to apply complex schema changes without any downtime.
• New dynamic UI (vtctld), rewritten from scratch in angular2.
• Update Stream functionality, for applications to subscribe to a change stream (for cache invalidation, for instance).
• Improved Map-Reduce support, for tables with non-uniform distributions.
• Increase large installation scalability with two-layer vtgate pools (l2vtgate, applicable to 100+ shard installations).
• Better Kubernetes support (Helm support, better scripts, ...).

Note that some of these features are still in development and not fully documented. This will change before we release the final 2.1 version.

Other noteworthy changes:

• Minimum required Go version changed to 1.7 (from 1.5).
• Updated to gRPC 1.0.0 (from 0.13.0).
• Added Percona Docker Images (changes contributed by Stitch Labs).

Vitess v2.0.0

Release to Manufacturing

This is the code cut for what will become the first GA release of Vitess 2.0.
We're now focused on updating documentation in preparation for General Availability.

Changes since v2.0.0-rc.2

Code Cleanup

• vtctld: Remove experimental VSchema editor UI

Vitess v2.0.0-rc.2

Changes since v2.0.0-rc.1

Bug Fixes

• Automatically create /zk root if it doesn't exist
• Fix edge cases in MariaDB backup/restore + reparent scenarios
• Randomize MySQL server-id when initializing tablet from scratch
  • Fixes edge case of replicate-same-server-id=0 together with backup/restore of former master.
• Fix for panic during resharding with unicode_loose_md5 vindex
• Fix for ApplySchema fails on DDLS which don't result in a schema change
• Fix for using SplitQuery with VTGate High-Level API
• VSchema for vtcombo is per-keyspace
  • This better matches the per-keyspace nature of VSchema in topology.
• Verify that autocommit is ON
  • This is required for proper operation of connection pooling.
• ApplySchema now reloads schema on slaves
  • Previously, only the master would immediately reload schema.
• Enable admin commands on vtcombo

Plugins

• Add backup storage plugin for Amazon S3

Vitess v2.0.0-rc.1

Changes since v2.0.0-beta.2

This is the first release candidate for v2.0.0. We are now freezing the release-2.0 branch for anything except bug fixes.

Client API

• Allow specifying a connection-wide default keyspace for VTGateV3 queries.
  (#1725,
  #1727,
  #1738)
  • Similar to the concept of a default database for a MySQL client connection.

SQL Support

• Preserve case of column names in query result field info.
• Preserve trailing comments in queries rewritten by VTGateV3.
• Support string-based hash vindexes
  • Previously, the value to hash had to be numeric or binary.

Management

• Hide VtTablet* and VtGate* commands behind a flag in vtctl/vtctld
  (#1700).
  • These commands are mostly meant for development, and should usually not be
    allowed in production.
  • Launch vtctld with the -enable_queries flag to allow these commands.
• Restart replication automatically
  if a tablet is restarted (e.g. as part of a Vitess upgrade).
• Optionally allow the use of insecure caller ID.
• It's no longer required to set a "sharding column name" when using VTGateV3.
  • i.e. no more vtctl SetKeyspaceShardingInfo when using VTGateV3.

Schema and VSchema

• VSchema config format changed
  • Some fields have been renamed for consistency.
• VSchema updates are now denormalized into per-cell records.
  (#1750)
  • No longer need to restart VTGate to pick up VSchema for new/removed
    keyspaces.
  • VSchema is still available if global topology goes down.
  • It's now possible to atomically move tables between keyspaces, for example
    when doing a vertical split.
  • NOTE: Even if you don't use VSchema, it's now required to run
    RebuildVSchemaGraph
    to initialize these records for a new keyspace.
• Schema changes are tested as a group, rather than individually.
  (#1749)
  • This allows batches of schema changes that depend on one another (#1731).

Interoperability

• Support MySQL 5.7
• Orchestrator Integration
  • Along with some patches contributed to Orchestrator, these new Vitess
    features allow the two systems to work together automatically.
    We'll be updating the documentation soon to show how to use it.

Code Cleanup

• Remove deprecated "serving graph" system, which had already been replaced by
  streaming health checks.
  • #1679,
    #1683,
    #1685,
    #1698,
    #1699,
    #1706
• Remove deprecated rowcache
  • This will be replaced by a caching scheme that makes sense for both VTGateV2
    and VTGateV3.
• Remove deprecated -target_tablet_type flag
  • Use -init_tablet_type instead.

Vitess v2.0.0-beta.2

Changes since v2.0.0-beta.1

Client Libraries

• JDBC-compatible wrapper
  • Contributed by Flipkart
  • See JDBC Example
• Update Java client to better match semantics of JDBC ResultSet
  • columnIndex now starts at 1
  • columnLabel is now case-insensitive
• Java: Synchronous client wrapper
  • To simplify and enforce proper usage of Java client in blocking mode.
• Java: Support more bind variable types
• Python: Always use colon style for bind variables
  • Removes extra translation layer for % formatting from Python client.
  • This makes the bind variable style consistent across all Vitess clients.
  • See local example client changes

SQL Support

• Phase 2 of VTGate V3 API
  • Can now route more complex queries.
  • Can perform cross-shard joins.
  • See phase 2 of V3 design for what is now supported.
• Cross-shard sequences
  • Note that NEXT is now a reserved word, and must be back-quoted if used as a column name.
• Support INTERVAL construct
• Support keyspace-qualified names
  • i.e. SELECT * FROM keyspace.table, analogous to db.table outside Vitess.
• Support STRAIGHT_JOIN and UNION DISTINCT SELECT

Resharding

• Support merging shards
• Support resharding tables with no keyspace_id column
  • As long as there is a VSchema that allows Vitess to compute the keyspace_id.
• SplitDiff: support custom sharding column name

Deployment and Management

• Configure SSL and authentication for gRPC
  • See Transport Security Model docs.
• VTGate watches for VSchema updates and refreshes automatically
• V3: Allow duplicate table names in vschema
  • As long as they're in different keyspaces.
• V3: Simplify VSchema
  • Remove the Class concept
  • VSchema is now per-keyspace
• Removed deprecated gorpc-based protocols
  • All protocols are now based on gRPC.
• Ceph Backup Storage Plugin
  • Contributed by Flipkart

Build Dependencies

• Pre-built Docker images for Percona Server
• We now depend on Go 1.5+ for native vendoring

Analysis

• Pre-scan algorithm for MapReduce/Hadoop query splitting

Vitess v2.0.0-beta.1

Changes since v2.0.0-alpha5

Client Libraries

• VTGateV3 Go driver
  can now also support certain types of VTGateV2 queries
• Publish PHP client to Packagist
• Refactor PHP client for better standards compliance
• PHP client returns associative array rows
• Java client is asynchronous by default
• Java client now returns 0 when converting SQL NULL to primitive types
• Python client now defaults to gRPC
• More sample clients for examples/local
  • In addition to Python,
    there are now sample clients for Go,
    PHP,
    and Java.

Query Service

• query.proto revamp
  • Fix NULL handling in new proto3-based query result format.
  • Unify type system for bind variables and query results.
• query_rules improvements
  • Improved JSON syntax for bindvar constraints.
  • Debug URL for viewing rule sets.
• Support REGEXP and RLIKE in
  SQL parser

Build Dependencies

• Default ZooKeeper version upgraded
  from 3.3.5 to 3.4.6.
• Update to gRPC 0.13
• Support building on OS X

Deployment and Management

• Add vtctld server to vtcombo
• Kubernetes config updates
  • Use Kubernetes DNS for service discovery
    and etcd bootstrap.
  • Kubernetes example for MySQL 5.6.
  • Use Kubernetes proxy to link from vtctld to vttablet status pages.
• Make MySQL users and privileges configurable
  when bootstrapping MySQL data directory.
  • Uses mysql_install_db and init_db.sql
    instead of extracting the old mysql-db-dir.tbz archive.
• Support semi-sync without async fallback
• Remove old vtctld UI files
  • Deployments no longer need to include go/cmd/vtctld/templates folder or
    set -templates flag on vtctld.
• Add option to bypass sanity check in ApplySchema
• Print human-readable tables for CLI queries
  • e.g. for ExecuteFetchAsDba

Vitess v2.0.0-alpha5

Changes since v2.0.0-alpha4

• New built-in backup
  plugin to store backups in Google Cloud Storage.
  • See the Vitess on Kubernetes guide for an
    example of usage.
• The Vitess PHP client now
  uses gRPC.
• Vitess Hadoop connector
  now uses the new gRPC-based Java client.
• Self-contained demo
  for VTGate V3, which is a preview feature as
  of Vitess 2.0.
• The vtctlclient-to-vtctld protocol now defaults to gRPC.

Vitess v2.0.0-alpha4

Changes since v2.0.0-alpha3

• Fixes for "bad node version" error when using etcd. (#1176, #1184)
• Java client is now feature-complete.
  • The last piece was our new canonical error code translation.

Vitess v2.0.0-alpha3

Changes since v2.0.0-alpha2

• Better PEP 0249 compliance in Python client.
• Type-checking result cursor for Java client.
• VTGate automatically adds annotations for filtered replication.
• All RPC protocols are now off by default, and must be explicitly enabled with the -service_map flag.
  • Previously the BSON protocols were enabled by default, and had to be explicitly disabled in -service_map if unwanted.
• SQL is now consistently represented as a UTF-8 string.
  • Previously it was represented in some places as a byte array.
• Update to gRPC beta.