Sijie,
Thank you for your reply!
I'll check it.
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
"Sijie Guo" <guosi...@gmail.com> wrote:
Yuri,
Thank you for bringing this up! This is a super helpful proposal!
The problem is very similar to what an RPC framework (like Finagle) with
client-side load balancing has.
An RPC framework with a client-side load-balancing mechanism needs to send
requests across multiple nodes. If you have an RPC service that has
thousands of nodes, there are thousands of clients connecting to that RPC
service. How to reduce the connections and how to effectively load balance
requests across thousands of nodes are the problems that a client-side
loading technology needs to solve. If you think about "partition" as "node"
and "partitioned producer" as "RPC client", the problem is exactly the
same. Finagle (the Twitter RPC framework) has implemented a lot of
client-side
load-balancing algorithms
<https://twitter.github.io/finagle/guide/Clients.html#load-balancing> and
there are some great articles that you can reference
<https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html>
.
I agree with the direction of introducing a mechanism to reduce the number
of producers in a partitioned topic producer. However, I have a concern
about introducing `.numPartitionsLimit(10)` directly to the producer
builder. It limits the possibility to implement different algorithms on
selecting partitions.
So instead of directly implementing the logic within the partitioned topic
producer, I think the proposal can be broken into two parts:
1) Introduce some kind of lazy-loading mechanism in the partitioned
producer to initialize the producers for partitions lazily. I.e., only
initialize a producer when the message router selects a partition.
2) Implement a message router that only selects one or N partitions.
In this way, the partitioned producer is only responsible for managing a
collection of producers, and the message router is responsible for
selecting the partitions. This allows people to be able to implement
different message routers. We can even adopt the client-side load balancing
algorithms from Finagle.
Thanks,
Sijie
On Wed, Jan 27, 2021 at 7:18 PM Yuri Mizushima <yumiz...@yahoo-corp.jp>
wrote:
> I notice that PIP-78 has already assigned to another issue.
>
>
https://mail-archives.apache.org/mod_mbox/pulsar-dev/202101.mbox/%3CCAG%3DTQOrPH49v9ToDE_aeQzEiDC%2BEgSR61ERoqanpWfQGvEB_Vw%40mail.gmail.com%3E
>
> So, I'll change the PIP number to 79.
>
>
https://github.com/apache/pulsar/wiki/PIP-79%3A-Reduce-redundant-producers-from-partitioned-producer
>
> Regards,
> --
> Yuri Mizushima
> yumiz...@yahoo-corp.jp
>
> "Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
>
> Dear Pulsar community,
>
> When partitioned producer connects to partitioned topic,
> sometimes doesn't need to connect to all of partitions depending on
> rate, routing mode, etc.
> So, I drafted a PIP about reducing redundant producers from
> partitioned producer.
> I'd like to use system resources (e.g. connections between Client and
> Broker, memory usage of both Client and Broker)
> more efficiently by this feature.
>
>
https://github.com/apache/pulsar/wiki/PIP-78%3A-Reduce-redundant-producers-from-partitioned-producer
>
> Feel free to ask me any questions or suggestions, etc.
>
> Best regards,
> --
> Yuri Mizushima
> yumiz...@yahoo-corp.jp
>
>
>
