RFC: Enhancements to Global Routing for Unsharded Tables

[rohit-nayak-ps]

Feature Description

Overview

This RFC proposes enhancements to the global routing mechanism in Vitess. The goal is to ensure
that unique tables from keyspaces without defined vschemas are globally routable. This document discusses the
current global routing features, the proposed changes, and provides examples to illustrate the impact of these changes.

Motivation

Addressing routing in tables

Vitess has two ways of addressing tables: using qualified names where the keyspace is specified or using unqualified
table names. Example: keyspace1.table1 vs. table1. Tables are currently only globally routable, i.e.
addressable using unqualified names, if

• there is only one keyspace in the cluster, which is unsharded, OR
• if there are multiple keyspaces and the tables are defined in the vschema for all keyspaces from which you
  want tables to be globally routable. Tables should also be unique across the vschemas.

Failure situation due to current logic

Second sharded keyspace

This logic recently caused down-time in a large production cluster. tl;dr: they were using global routing with a
single keyspace and then moved some tables to a second sharded keyspace, without specifying a VSchema to the unsharded
one.

Detailed Steps for this scenario:

• User has a single unsharded keyspace unsharded1 and is using unqualified table names, because their app had been
  written using unqualified names while targeting a regular MySQL DB. vtgate correctly routes this because there is
  only one unsharded keyspace: there is no vschema to consult because it was not necessary at this point to create one
  for the unsharded keyspace.
• User wants to reshard some large tables. Say, A MoveTables workflow is started into a sharded
  keyspace sharded1, which has a vschema with all tables/vindexes defined. Say table2 is moved in this
  workflow but table1 continues to live in unsharded1
• Thus tables with the same name table2 exist in both the user's DB as well as in sharded1. Routing rules have
  already been setup so that the unqualified tables are routed to unsharded1. sharded1 is still not-serving, so
  the tables in sharded1 are not visible to vtgate.
• When the MoveTablesworflow has caught up and the user does a SwitchWrites (i.e.SwitchTraffic for primary)
  sharded1 is now serving, but routing rules are updated to point to sharded, so global routing for table2
  still works.
• The problem arises when user does a Complete on the workflow to clean things up, using --keep-data to keep the
  original tables around. Routing rules now get deleted.
• vtgate now finds table2 in both keyspaces and the queries using table2 start failing with an ambiguous table error.

Second unsharded keyspace

This issue is due to (arguably bugs in) the current logic.

• A cluster starts with a single unsharded keyspace, without a VSchema, using global routing.
• A second unsharded keyspace is added, also without a VSchema. This keyspace has no common tables.
• Now that there is more than one keyspace, none of the tables are routable. Queries using global routing now
  stop working with a "table not found" error.

Current Global Routing Implementation

Global Routable Tables

In Vitess, a table is considered globally routable if it meets the following criteria:

• There is only one unsharded keyspace, which has that table.
• There are multiple keyspaces and it is present in one (and only one) VSchema.

Ambiguous Tables

An unqualified table is considered ambiguous if the table exists in two or more VSchemas.

Unrecognized Tables

For clusters with multiple keyspaces, once any of them has a VSchema only the unique tables across VSchemas are
globally addressable. Tables in keyspaces without a VSchema are not routable: these return a table not found error.

Example

Consider the following keyspaces and tables:

• keyspace1 (unsharded): table1, table2 with VSchema
• keyspace2 (unsharded): table2, table3 with no VSchema
• keyspace3 (sharded): table2, table4 with VSchema

In this scenario, for unqualified access:

• table1 is routable.
• table2 is ambiguous
• table3 is not routable
• table4 is routable.

Proposed Changes

The proposed changes aim to make tables from keyspaces without defined vschemas globally routable. Specifically, the
changes include:

Automatic Inclusion of Tables from Keyspaces without Vschemas

Tables from keyspaces that do not have vschemas defined will be automatically included in the global routing table.

Conflict Resolution

Tables in defined vschemas have priority. If one of the keyspaces without vschema has a table of the same name then
those can only be addressed qualified. If those tables are globally addressed, they will be routed to the keyspace
in the vschema.

Tables which are not unique in the keyspaces with no vschemas defined will be marked as ambiguous.

Example

Consider the following keyspaces and tables after the proposed changes:

• keyspace1 (unsharded, no vschema): table1, table2 with VSchema
• keyspace2 (unsharded, no vschema): table2, table3 with no VSchema
• keyspace3 (sharded): table4 with VSchema

In this scenario, for unqualified access:

• table1 is routable.
• table2 is ambiguous
• table3 is routable. Improvement from current logic
• table4 is routable

Benefits

• Improved Global Routing: Ensures that tables from keyspaces without vschemas are included in the global routing
  table, preventing hard-down situations as detailed above.
• Simplified Configuration: Reduces the need for explicit vschema definitions for unsharded keyspaces.

Use Case(s)

.