Message was moderated onto the list, including the OP Sent from my iPhone
> On Nov 12, 2018, at 2:28 AM, Ivan Kelly <iv...@apache.org> wrote: > > Hi Koushik, > >> I just got introduced to Apache pulsar and studying it to solve a problem at >> hand. >> As I understand that Pulsar supports 1M of topics, Is there any limitation >> on no.of producers and subscribers that brokers can handle? I have the >> following use cases. > > The limit of how many producers and consumers a broker can serve is a > function of the network bandwidth and CPU of that broker. However, the > way Pulsar is designed, if the number of producers/consumers becomes > too large for the number of brokers you have, you can add broker > nodes, and some topics will move to the new nodes, balancing your load > on all the machines. > > In terms of consumers, depending on which kind of consumer you use, > these may count towards the 1M topic limit. The real limit in pulsar > is the amount of metadata that can be stored in zookeeper. Everything > else can be scaled horizontally. Each topic has one piece of metadata > that belongs to the topic itself. However, each durable subscription > (as is the default) also has one piece of metadata. So in the examples > you give below, if you are creating consumers using > PulsarClient#newSubscriber(), you will have ~200k pieces of metadata > stored in ZK. ZK can easily hold 200k pieces of metadata. I think the > highest I've seen it go is 2M pieces of metadata. > >> Usecase 1 : Few hundred producers producing to 100k topics, and each topic >> has corresponding consumer (i.e 100k consumers) trying to consume messages >> from its topic. To summarize ~100k clients interacting with brokers here. >> >> Usecase 2 : 100k producing to corresponding topic (i.e 100k topics), and >> each topic has corresponding consumer (i.e 100k consumers) trying to consume >> messages from its topic. To summarize ~200k clients interacting with brokers >> here. > > Both these usecases look very doable with Pulsar. > > Regards, > Ivan
