Hi Koushik, > I just got introduced to Apache pulsar and studying it to solve a problem at > hand. > As I understand that Pulsar supports 1M of topics, Is there any limitation on > no.of producers and subscribers that brokers can handle? I have the following > use cases.
The limit of how many producers and consumers a broker can serve is a function of the network bandwidth and CPU of that broker. However, the way Pulsar is designed, if the number of producers/consumers becomes too large for the number of brokers you have, you can add broker nodes, and some topics will move to the new nodes, balancing your load on all the machines. In terms of consumers, depending on which kind of consumer you use, these may count towards the 1M topic limit. The real limit in pulsar is the amount of metadata that can be stored in zookeeper. Everything else can be scaled horizontally. Each topic has one piece of metadata that belongs to the topic itself. However, each durable subscription (as is the default) also has one piece of metadata. So in the examples you give below, if you are creating consumers using PulsarClient#newSubscriber(), you will have ~200k pieces of metadata stored in ZK. ZK can easily hold 200k pieces of metadata. I think the highest I've seen it go is 2M pieces of metadata. > Usecase 1 : Few hundred producers producing to 100k topics, and each topic > has corresponding consumer (i.e 100k consumers) trying to consume messages > from its topic. To summarize ~100k clients interacting with brokers here. > > Usecase 2 : 100k producing to corresponding topic (i.e 100k topics), and > each topic has corresponding consumer (i.e 100k consumers) trying to consume > messages from its topic. To summarize ~200k clients interacting with brokers > here. Both these usecases look very doable with Pulsar. Regards, Ivan
