PIP 8: Pulsar beyond 1M topics
- Status: Implemented
- Pull Request: #903
Pulsar was designed to serve 1M topics in a cluster. We have a need to scale Pulsar beyond 1M topics. Recent improvements have stretched this 1M limit somewhat, but Pulsar startup time from a Cloud state with 1M+ topics starts to increase with number of topics. This proposal is to scale beyond 1M+ topics, and still keep a a limit on startup time
- Must not require a client upgrade (Pulsar wire protocol doesn’t change)
- Must be backward compatible
- Must provide a painless upgrade
This document describes a high level design for extending Pulsar “namespace” space (hereafter termed NS space) across more than one Pulsar cluster. The Global NS space of a Pulsar cluster is disambiguated using ZK paths in the global ZK. Topics are not stored in Global ZK. Pulsar can support tens of thousands of NS paths, and the Global ZK can scale to a that many thousands of namespaces. If each NS can support 1-1000 topics, Pulsar can reach 10M topics.
If the NS space can be partitioned across multiple Pulsar clusters (effectively more local ZKs) , then the clusters can share the topic load. With the abstraction of a single NS space across multiple Pulsar clusters, Pulsar then scales to multiple millions of topics.
Each region has one Pulsar cluster. One Global ZK, one local ZK, along with the brokers and bookies form a Pulsar cluster.
The new architecture will have more than one Pulsar cluster in each region. The figure below is an example with Pulsar across 2 regions, with 2 local clusters in each.
The Global NS space is unique for global namespaces, so this does not require any special consideration. The only problem to be resolved is how the NS space is to be distributed among a particular group of local clusters. The NS space assignment needs to deterministic and stateful within a cluster. If that can be achieved, all existing mechanisms of administration and operation would continue to work. The local clusters in effect become distinct clusters with no interaction, except than using the Global ZK to ensure there are no collisions in the NS space across them. Fortunately, there is no need to make any changes to Pulsar to make the NS assignment stateful and deterministic.
We here introduce the concept of a peer group (peers). This is the group of clusters that together hosts an instance of the Global NS space.
Peer groups are defined and set in the Global ZK. Each local cluster has its own defined peer group, which is set administratively.
Namespaces are created as usual, and results in an update to the Global ZK. The namespace node gets created in the Global ZK. Replication clusters are also set as usual, for eg, C3 and C4 for NS2. This is the local clusters where the namespace resides, and the topics in that namespace are created in the specific local ZK for those clusters. So far, this is nothing different from existing functionality.
A Namespace resides on specific local clusters, as identified by its replication set, Provisioning tenants and Namespaces will still need to be done on specific local clusters
To maintain the abstraction of a single cluster, a local cluster needs to be smart enough to know the peers which share the Global NS space with it. When a local cluster receives a lookup request, the local cluster looks up the namespace, to see if it is present in the Global ZK. The receiving cluster then sees if it is the owner of that namespace. If it is, then the lookup is completed in the local cluster. If not, it then redirects it to the next local cluster in the peer group, (and so on…), with the appropriate hints to break lookup cycles.
With the concept of a peer group and the implementation of cluster redirect in lookups, the NS space can be spread across more than one local Pulsar cluster Lookup URLs The peer group as a whole needs to get a new URL, which fronts the entire list of brokers of the peer group. The local clusters will need local lookup URLs which are needed for inter-peer group cluster redirects.
Replication within a peer group will be disallowed, as it violates the namespace uniqueness requirement. Attempting to do so will fail automatically if the local clusters share the same Global ZK installation. Otherwise it will have to be enforced at the API implementation level. The examples above show a Global ZK installation for each local cluster, but that is not necessary.
Nothing in this design requires that we have an identical regions of Pulsar clusters across all datacenters. For eg: We could have one local cluster in one datacentre, and 2 in another.
This is the scenario where Pulsar is running in a single local cluster mode, and needs to add a new local cluster to it.
We will prefer that replication URLs be the local cluster URLs. So the existing URLs will need to remain as local cluster URLs.
This means that we will need to create a new peer group URL.
Existing clients will then need to update their URL settings to the peer group URL. During the upgrade period, both URLs, - the new peer group URL and the existing local cluster URL - will front end the same group of brokers; the existing cluster brokers. After the clients change the setting to the peer group URL, the peer group URL can front-end both set of brokers, from the existing local cluster and the new cluster.
Strictly speaking, this client setting change is not required; the only drawback to not doing this is that all lookups will always end up in the existing local cluster.
If each region has multiple pulsar clusters setup then we can configure peer-clusters that exist into same region.
eg:
1. Setup Pulsar clusters in each region:
We have two regions us-west and us-east, and each region has 3 pulsar clusters setup.
Pulsar clusters setup in us-west region: us-west-1, us-west-2, us-west-3
bin/pulsar-admin clusters create us-west-1 --url http://pulsar.us-west-1.com
bin/pulsar-admin clusters create us-west-2 --url http://pulsar.us-west-2.com
bin/pulsar-admin clusters create us-west-3 --url http://pulsar.us-west-3.com
Pulsar clusters setup in us-east region: us-east-1, us-east-2, us-east-3
bin/pulsar-admin clusters create us-east-1 --url http://pulsar.us-east-1.com
bin/pulsar-admin clusters create us-east-2 --url http://pulsar.us-east-2.com
bin/pulsar-admin clusters create us-east-3 --url http://pulsar.us-east-3.com
2. Configure peer-clusters which have been setup in the same region
a. Setup us-west-2 and us-west-3 as peer-clusters of us-west-1. So, if us-west-1 receives lookup request for namespace that would be owned by any peer-cluster then us-west-1 can redirect that lookup request to appropriate cluster.
bin/pulsar-admin clusters update-peer-clusters us-west-1 --peer-clusters us-west-2,us-west-3
b. Setup us-east-2 and us-east-3 as peer-clusters of us-east-1.
bin/pulsar-admin clusters update-peer-clusters us-east-1 --peer-clusters us-east-2,us-east-3
3. Create namespace with geo-replication enabled
bin/pulsar-admin namespaces create sample/global-ns -c us-west-3,us-east-3
Now, if us-west-1 receives lookup request for namespace sample/global-ns then us-west-1 knows about its peer-cluster us-west-3 and it redirects lookup to us-west-3 so, us-west-3 can serve all the topics under that namespace. Same applies to us-east peer-clusters as well.