Do you have any comments?
If there are no comments by Dec. 7, I will close the discussion and rebase the
PR commit to current master.
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
On 2021/11/16 15:46, "Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
Dear Pulsar community,
I have created a new PR
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Sn57dCx1iItNEMArKWAXxE9frCbmQClw%2Fts0nmMkwyw%3D&reserved=0
for stats aggregation,
but I didn't discuss about the wire protocol change. I hope we will discuss
it here.
Currently, partitioned producer can't aggregate by any key such as cnx,
producerId, producerName, and so on.
I think we need to add any aggregation system.
Therefore, added new aggregation policy as producerName (with client side
implementation).
New protocol field partial_producer_supported is not used for stats
aggregation. It is used for backward compatibility.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QswqgZHSeHJArmgAdBx1wPrm6rX4l4lTf39hs57g6NI%3D&reserved=0
In my understanding, if introduce new stats aggregation key to client side,
need a way to determine whether the feature is enabled at client side.
For example, whether the producer has specific field or metadata,
the version (e.g. protocol version) is greater than threshold, etc.
Of course, if we can introduce aggregation feature without adding any new
key or implementations from client side,
we can support the feature not only new client but also old one.
I'm looking forward to your opinions or suggestions to this PR.
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
On 2021/05/11 14:26, "Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
> I will submit the next PR about PartitionedTopicStats later.
I submitted the next PR for this PIP. If you have any suggestions, please
comment to this PR.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10534&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cHqzaY1H4N%2FaQEemy5ZRQSBAiXa3FDXDuFLyoJ4dRtA%3D&reserved=0
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
"Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
Dear Pulsar Community,
I submitted the PR for this PIP.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F10279&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AzmPY98arKmn7UM1%2F6FJMwOKOavPVC5xVG%2Br5FBJlH4%3D&reserved=0
This is a part of implementations.
I will submit the next PR about PartitionedTopicStats later.
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
"Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
Sijie,
After sending previous mail, I watched meeting recording and
understand about authn/authz issue.
Therefore, I updated the PIP document.
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lZrLE4EhYfy7xUNmfUraIchYOkH%2B0HQaeVcKGaTBWLw%3D&reserved=0
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
"Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
Sijie,
> If the lazy-loading approach sounds attractive to you and you
like it,
> maybe the next step is to update the PIP, what do you think?
I think so too. I will update the PIP after discussing the
authn/authz issue.
Regards,
--
Yuri Mizushima
yumiz...@yahoo-corp.jp
"Sijie Guo" <guosi...@gmail.com> wrote:
Hi Yuri,
Regarding the authn/authz issue, @Matteo Merli
<mme...@apache.org> can
probably chime in more about that part.
If the lazy-loading approach sounds attractive to you and
you like it,
maybe the next step is to update the PIP, what do you think?
- Sijie
On Mon, Feb 8, 2021 at 6:57 PM Yuri Mizushima
<yumiz...@yahoo-corp.jp>
wrote:
> Michael,
>
> Thank you for your comment!
>
> > Which Pulsar Clients will benefit from this proposal?
> I think that this proposal will be useful to any clients.
> In my schedule, if this proposal is accepted then I will
implement this
> feature to Java client.
> If needed, then implement same feature to other clients
such as C++, Go,
> etc.
>
> Regards,
> --
> Yuri Mizushima
> yumiz...@yahoo-corp.jp
>
>
> "Michael Marshall" <mikemars...@gmail.com> wrote:
>
> Hi Yuri and Sijie,
>
> I definitely like the idea of lazily creating
producers as well as
> introducing a way to provide custom routing logic.
>
> Which Pulsar Clients will benefit from this proposal?
I’d love to see
> this feature in the go client.
>
> Thanks,
> Michael Marshall
>
> > On Feb 7, 2021, at 9:53 PM, Yuri Mizushima
<yumiz...@yahoo-corp.jp>
> wrote:
> >
> > Sijie,
> >
> > Thank you for sharing!
> >
> > First, I considered your suggestion.
> > I think these implementations sound good.
> >
> > I think we should consider the State of partitioned
producer: Ready,
> Connecting, etc.
> > Currently, partitioned producer gets "Ready" only
when all producers
> connect to Broker correctly.
> >
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPartitionedProducerImpl.java%23L146&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKgraLiuHSkiE9un5L6sWeS8PQLHLkM9nxiqGWt%2FFhw%3D&reserved=0
> > It seems that we should change meaning of state (or
change handling)
> if we introduce the lazy-load feature.
> > To guarantee the message ordering (e.g. using
partitionKey),
> partitioned producer should stop (or don't send messages
to be routed to
> unavailable partition) when producer can't connect to one
of partition.
> >
> > Secondly, I considered Matteo's comments.
> > I couldn't understand well about issue of
authn/authz. Please tell
> me more detail.
> >
> > I wrote "connection" as number of producers which
connect to broker.
> Also, TCP connections between partitioned producer and
broker will be less
> than or equal to current in some cases. I'll show a case
below.
> >
> > Suppose
> > * cluster has Broker0, 1, 2
> > * partitioned topic has 5 partitions
> > * limit conf is 3 partitions
> > * loadbalance partitions as below
> > - Broker0: partition-0, partition-1
> > - Broker1: partition-2
> > - Broker2: partition-3, partition-4
> >
> > Currently, client will create 3 connections
(Broker0, 1, 2). If
> client uses limit conf and elects partitions such as [0,
1, 2], then client
> will create 2 connections (Broker0, 1). Of course, if
client elects
> partitions such as [0, 2, 3], then client will still
create 3 connections.
> >
> > I'd like to decrease number of producers. I think
that resources of
> broker will be improved slightly by this feature because
broker has list of
> producers by some classes such as ServerCnx,
AbstractTopic.
> >
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FServerCnx.java%23L1096-L1097&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rx3Ov4r4dlL5YUpt4Bbl%2FhJGpmoeXNsMmS33TDcYrF8%3D&reserved=0
> >
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2Ffa41d02bebfd841767846240f3ae574047f118f0%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractTopic.java%23L577&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802266126%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hx910MdAyt9jz7CfcYMxRuRFbkOJKUGe8mrpb67JmqA%3D&reserved=0
> >
> > In my case, unspecified number of producers will
connect to the same
> partitioned topic with different rate. We need to set the
number of
> partitions according to the high-rate producer.
> > However, on the other hand, this number is
excessively large for
> low-rate producers.
> > I want to reduce such redundant producers for
resource efficiency.
> >
> > Regards,
> > --
> > Yuri Mizushima
> > yumiz...@yahoo-corp.jp
> >
> >
> > "Sijie Guo" <guosi...@gmail.com> wrote:
> >
> > Hi Yuri,
> >
> > In today's community meeting, Matteo shared some
of his thoughts
> about this
> > PIP.
> >
> > You can find some meeting notes here:
> >
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE%2Fedit%23bookmark%3Did.rezbt4xmjxpz&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f0bK%2Bzbd2NnxCgVLTYty%2BeIftpfM0%2BpBFC%2FySEBARoo%3D&reserved=0
> >
> > Matteo can also chime in as well.
> >
> > - Sijie
> >
> >> On Sun, Jan 31, 2021 at 7:21 PM Yuri Mizushima <
> yumiz...@yahoo-corp.jp>
> >> wrote:
> >> Sijie,
> >> Thank you for your reply!
> >> I'll check it.
> >> Regards,
> >> --
> >> Yuri Mizushima
> >> yumiz...@yahoo-corp.jp
> >> "Sijie Guo" <guosi...@gmail.com> wrote:
> >> Yuri,
> >> Thank you for bringing this up! This is a super
helpful proposal!
> >> The problem is very similar to what an RPC
framework (like Finagle)
> >> with
> >> client-side load balancing has.
> >> An RPC framework with a client-side
load-balancing mechanism needs
> to
> >> send
> >> requests across multiple nodes. If you have an
RPC service that has
> >> thousands of nodes, there are thousands of
clients connecting to
> that
> >> RPC
> >> service. How to reduce the connections and how to
effectively load
> >> balance
> >> requests across thousands of nodes are the
problems that a
> client-side
> >> loading technology needs to solve. If you think
about "partition"
> as
> >> "node"
> >> and "partitioned producer" as "RPC client", the
problem is exactly
> the
> >> same. Finagle (the Twitter RPC framework) has
implemented a lot of
> >> client-side
> >> load-balancing algorithms
> >> <
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.github.io%2Ffinagle%2Fguide%2FClients.html%23load-balancing&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FeyGeIXYMkZ9dEtEklKd3TEbN0rst2tyIEVAznbidXs%3D&reserved=0>
> >> and
> >> there are some great articles that you can
reference
> >> <
> >>
>
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/daperture-load-balancer.html
> >> .
> >> I agree with the direction of introducing a
mechanism to reduce the
> >> number
> >> of producers in a partitioned topic producer.
However, I have a
> concern
> >> about introducing `.numPartitionsLimit(10)`
directly to the
> producer
> >> builder. It limits the possibility to implement
different
> algorithms on
> >> selecting partitions.
> >> So instead of directly implementing the logic
within the
> partitioned
> >> topic
> >> producer, I think the proposal can be broken into
two parts:
> >> 1) Introduce some kind of lazy-loading mechanism
in the partitioned
> >> producer to initialize the producers for
partitions lazily. I.e.,
> only
> >> initialize a producer when the message router
selects a partition.
> >> 2) Implement a message router that only selects
one or N
> partitions.
> >> In this way, the partitioned producer is only
responsible for
> managing
> >> a
> >> collection of producers, and the message router
is responsible for
> >> selecting the partitions. This allows people to
be able to
> implement
> >> different message routers. We can even adopt the
client-side load
> >> balancing
> >> algorithms from Finagle.
> >> Thanks,
> >> Sijie
> >> On Wed, Jan 27, 2021 at 7:18 PM Yuri Mizushima <
> yumiz...@yahoo-corp.jp
> >> wrote:
> >>> I notice that PIP-78 has already assigned to
another issue.
> >>
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fpulsar-dev%2F202101.mbox%2F%253CCAG%253DTQOrPH49v9ToDE_aeQzEiDC%252BEgSR61ERoqanpWfQGvEB_Vw%2540mail.gmail.com%253E&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKRnaunNSAtXSWwCGayzd26SXG%2F6oKg15n%2BY1fUrZHA%3D&reserved=0
> >>> So, I'll change the PIP number to 79.
> >>
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-79%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EVNZADVJ8nsOwi8mS3sErIVkHJ0Cc8LXVlaIQEG0BbU%3D&reserved=0
> >>> Regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumiz...@yahoo-corp.jp
> >>> "Yuri Mizushima" <yumiz...@yahoo-corp.jp> wrote:
> >>> Dear Pulsar community,
> >>> When partitioned producer connects to
partitioned topic,
> >>> sometimes doesn't need to connect to all of
partitions depending
> >> on
> >>> rate, routing mode, etc.
> >>> So, I drafted a PIP about reducing redundant
producers from
> >>> partitioned producer.
> >>> I'd like to use system resources (e.g.
connections between
> >> Client and
> >>> Broker, memory usage of both Client and Broker)
> >>> more efficiently by this feature.
> >>
>
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fwiki%2FPIP-78%253A-Reduce-redundant-producers-from-partitioned-producer&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Cd1cf3f01dc4b48994a0008d9a8cc7154%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637726419802276119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ncPm1G5ZCCtEfvR682J%2FYHH42G5Ns5eVJn7j%2BE40hR8%3D&reserved=0
> >>> Feel free to ask me any questions or
suggestions, etc.
> >>> Best regards,
> >>> --
> >>> Yuri Mizushima
> >>> yumiz...@yahoo-corp.jp
>
>
