>>The real limit in pulsar is the amount of metadata that can be stored in zookeeper.
After, PIP-8, It is now possible to run multiple Pulsar clusters under one service URL (going beyond the limits of a what a single ZK can support). Joe On Mon, Nov 12, 2018 at 7:36 AM, Dave Fisher <dave2w...@comcast.net> wrote: > Message was moderated onto the list, including the OP > > Sent from my iPhone > > > On Nov 12, 2018, at 2:28 AM, Ivan Kelly <iv...@apache.org> wrote: > > > > Hi Koushik, > > > >> I just got introduced to Apache pulsar and studying it to solve a > problem at hand. > >> As I understand that Pulsar supports 1M of topics, Is there any > limitation on no.of producers and subscribers that brokers can handle? I > have the following use cases. > > > > The limit of how many producers and consumers a broker can serve is a > > function of the network bandwidth and CPU of that broker. However, the > > way Pulsar is designed, if the number of producers/consumers becomes > > too large for the number of brokers you have, you can add broker > > nodes, and some topics will move to the new nodes, balancing your load > > on all the machines. > > > > In terms of consumers, depending on which kind of consumer you use, > > these may count towards the 1M topic limit. The real limit in pulsar > > is the amount of metadata that can be stored in zookeeper. Everything > > else can be scaled horizontally. Each topic has one piece of metadata > > that belongs to the topic itself. However, each durable subscription > > (as is the default) also has one piece of metadata. So in the examples > > you give below, if you are creating consumers using > > PulsarClient#newSubscriber(), you will have ~200k pieces of metadata > > stored in ZK. ZK can easily hold 200k pieces of metadata. I think the > > highest I've seen it go is 2M pieces of metadata. > > > >> Usecase 1 : Few hundred producers producing to 100k topics, and each > topic has corresponding consumer (i.e 100k consumers) trying to consume > messages from its topic. To summarize ~100k clients interacting with > brokers here. > >> > >> Usecase 2 : 100k producing to corresponding topic (i.e 100k topics), > and each topic has corresponding consumer (i.e 100k consumers) trying to > consume messages from its topic. To summarize ~200k clients interacting > with brokers here. > > > > Both these usecases look very doable with Pulsar. > > > > Regards, > > Ivan > >
