Re: [DISCUSS] PIP-250: Add proxyVersion to CommandConnect

[Xiangying Meng]

Hi Michael,

I appreciate your detailed explanation.
I agree with your proposal.
It seems that sharing version information between the proxy
and broker would indeed be beneficial for debugging purposes without
creating
 an unnecessary tight coupling between the two components.

I appreciate your willingness to discuss this feature further
r and for taking the time to address the concerns raised.

Best regards,
Xiangying

On Tue, Apr 11, 2023 at 12:17 PM Michael Marshall 
wrote:

> Thanks for your feedback Mattison and Xiangying. I'll note that the
> PIP vote did close already and I have the implementation just about
> done, but I'm happy to discuss this feature a bit more here.
>
> > we should avoid coupling the concepts in the proxy and the broker
>
> Sharing version information does not tightly couple the proxy and the
> broker. It makes it easier to collect the relevant debugging
> information that can help solve problems faster. The client already
> tells the broker its version. Are you able to provide more insight
> here?
>
> > The proxy should be a separate component. Instead of continuing to
> couple the relevant proxy concepts into the broker, everyone should be a
> client to the broker.
>
> I don't think this analogy holds true. The pulsar proxy is not a layer
> 4 proxy. It sends its own pulsar protocol messages. I want to quickly
> know the proxy version when debugging issues observed in the broker,
> and this is the only way I see for the broker to get this information.
>
> > I think the intention behind this is excellent, but directly modifying
> the protocol might be a bit too heavy-handed.
>
> Do you disagree with sharing this information with the broker? I think
> we generally view the protocol as lightweight.
>
> > Wouldn't it be better to directly expose the proxyVersion and
> clientVersion information via Prometheus metrics
>
> Which server is producing these metrics? My goal is for the broker to
> get this information so it can be part of the logs. The only way to
> transport these metrics is via the protocol.
>
> If there is a serious objection, I can add an option for the proxy to
> withhold this data from the broker. However, I think we should enable
> it by default.
>
> Thanks,
> Michael
>
> On Sat, Apr 8, 2023 at 2:19 AM Xiangying Meng 
> wrote:
> >
> > Dear Michael,
> >
> > I think the intention behind this is excellent, but directly modifying
> the protocol might be a bit too heavy-handed.
> >  This approach may lead to data redundancy.
> >  In large-scale clusters, every client connection would need to transmit
> the extra proxy version information,
> > which might increase network overhead.
> > Therefore, we should consider a more lightweight solution.
> >
> > If the primary purpose of the proxy version information is for
> diagnostics or auditing,
> > we could explore alternative methods for collecting this information
> without modifying the protocol level.
> > Wouldn't it be better to directly expose the proxyVersion and
> clientVersion information via Prometheus metrics,
> > as mentioned in the "Future work(Anything else)`" section of the
> proposal?
> >
> > Please let me know what you think about this suggestion.
> >
> > Best regards,
> > Xiangying
> >
> > On Thu, Apr 6, 2023 at 3:46 PM  wrote:
> >>
> >> Sorry for the late response.
> >>
> >> Why do we need to make the broker aware of the proxy when, by normal
> software design, we should avoid coupling the concepts in the proxy and the
> broker? The previous authentication was for historical reasons, but we
> should not continue to introduce this coupling.
> >>
> >> The proxy should be a separate component. Instead of continuing to
> couple the relevant proxy concepts into the broker, everyone should be a
> client to the broker.
> >>
> >> Best,
> >> Mattison
> >> On Feb 25, 2023, 01:12 +0800, Michael Marshall ,
> wrote:
> >> > Great suggestions, Enrico.
> >> >
> >> > > It would be interesting to reject connections on the Proxy if the
> >> > > client tries to set that field.
> >> >
> >> > I support making the proxy reject invalid input. We could also have
> >> > the client reject connections where the client connect command
> >> > includes `original_principal`, `original_auth_data`, or
> >> > `original_auth_method`, since those are also only meant to be sent by
> >> > the proxy.
> >> >
> >> > > On the broker there is no way to distinguish a proxy from a client,
> that's fair.
> >> >
> >> > We can reject these connections when authentication and authorization
> >> > are enabled. My draft PR includes such logic.
> >> >
> >> > Thanks,
> >> > Michael
> >> >
> >> > On Fri, Feb 24, 2023 at 7:29 AM Enrico Olivelli 
> wrote:
> >> > >
> >> > > Makes sense.
> >> > >
> >> > > It would be interesting to reject connections on the Proxy if the
> >> > > client tries to set that field.
> >> > > On the broker there is no way to distinguish a proxy from a client,
> that's fair.
> >> > > But on the proxy it is not expected to see a connection

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

[Lin Lin]

> Do we expect that the bundles should have the same loads?

Yes.If the bundle load is similar, it will be easier to achieve a balanced 
state during subsequent load balancing.
If the bundle loads are different or there are hot bundles, it will trigger 
bundle split or unload.
Unload the hotspot bundle can easily make the Broker receiving it become a new 
hotspot.
The example you gave is very ideal. In the actual production environment, 
the load difference between bundles is very large, 
and the existing load balancing algorithm is difficult to achieve balance.


> Can it be resolved by introducing the entry write/read rate to the bundle 
> stats?

The implementation of ThresholdSheder needs to set the threshold. 
Even if the read/write rate is added, the threshold also needs to be set. 
It needs to be set according to different scenarios, which is very inconvenient 
to use.

> Can we try to force-sync the load data of the brokers before performing the 
> distribution of a large number of bundles?

After the bundle is assigned to new Broker, it needs to run for a period of 
time to reflect the new load, 
so mandatory reporting is not very useful.

> IMO, the proposal should clearly describe the Goal, like which problem will 
> be resolved with this proposal.

I have re-added the Goal and the limit of this PIP in issue.
For me, if the community doesn't need my implementation,
it is fine to only plug-in the assignment algorithm.

On 2023/04/10 12:23:03 PengHui Li wrote:
> Hi Lin,
> 
> > The load managed by each Bundle is not even. Even if the number of
> partitions managed
>by each bundle is the same, there is no guarantee that the sum of the
> loads of these partitions
>will be the same.
> 
> Do we expect that the bundles should have the same loads? The bundle is the
> base unit of the
> load balancer, we can set the high watermark of the bundle, e.g., the
> maximum topics and throughput.
> But the bundle can have different real loads, and if one bundle runs out of
> the high watermark, the bundle
> will be split. Users can tune the high watermark to distribute the loads
> evenly across brokers.
> 
> For example, there are 4 bundles with loads 1, 3, 2, 4, the maximum load of
> a bundle is 5 and 2 brokers.
> We can assign bundle 0 and bundle 3 to broker-0 and bundle 1 and bundle 2
> to broker-2.
> 
> Of course, this is the ideal situation. If bundle 0 has been assigned to
> broker-0 and bundle 1 has been
> assigned to broker-1. Now, bundle 2 will go to broker 1, and bundle 3 will
> go to broker 1. The loads for each
> broker are 3 and 7. Dynamic programming can help to find an optimized
> solution with more bundle unloads.
> 
> So, should we design the bundle to have even loads? It is difficult to
> achieve in reality. And the proposal
> said, "Let each bundle carry the same load as possible". Is it the correct
> direction for the load balancer?
> 
> > Doesn't shed loads very well. The existing default policy
> ThresholdShedder has a relatively high usage
>threshold, and various traffic thresholds need to be set. Many clusters
> with high TPS and small message
>bodies may have high CPU but low traffic; And for many small-scale
> clusters, the threshold needs to be
>modified according to the actual business.
> 
> Can it be resolved by introducing the entry write/read rate to the bundle
> stats?
> 
> > The removed Bundle cannot be well distributed to other Brokers. The load
> information of each Broker
>will be reported at regular intervals, so the judgment of the Leader
> Broker when allocating Bundles cannot
>be guaranteed to be completely correct. Secondly, if there are a large
> number of Bundles to be redistributed,
>the Leader may make the low-load Broker a new high-load node when the
> load information is not up-to-date.
> 
> Can we try to force-sync the load data of the brokers before performing the
> distribution of a large number of
> bundles?
> 
> For the Goal section in the proposal. It looks like it doesn't map to the
> issues mentioned in the Motivation section.
> IMO, the proposal should clearly describe the Goal, like which problem will
> be resolved with this proposal.
> Both of the above 3 issues or part of them. And what is the high-level
> solution to resolve the issue,
> and what are the pros and cons compared with the existing solution without
> diving into the implementation section.
> 
> Another consideration is the default max bundles of a namespace is 128. I
> don't think the common cases that need
> to set 128 partitions for a topic. If the partitions < the bundle's count,
> will the new solution basically be equivalent to
> the current way?
> 
> If this is not a general solution for common scenarios. I support making
> the topic-bundle assigner pluggable without
> introducing the implementation to the Pulsar repo. Users can implement
> their own assigner based on the business
> requirement. Pulsar's general solution may not be good for all scenarios,

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

[Xiangying Meng]

Hi Linlin,
> This is an incompatible modification, so the entire cluster needs to be
upgraded, not just a part of the nodes

Appreciate your contribution to the new feature in PIP-255.
 I have a question regarding the load-balancing aspect of this feature.

You mentioned that this is an incompatible modification,
and the entire cluster needs to be upgraded, not just a part of the nodes.
 I was wondering why we can only have one load-balancing strategy.
Would it be possible to abstract the logic here and make it an optional
choice?
This way, we could have multiple load-balancing strategies,
such as hash-based, round-robin, etc., available for users to choose from.

I'd love to hear your thoughts on this.

Best regards,
Xiangying

On Mon, Apr 10, 2023 at 8:23 PM PengHui Li  wrote:

> Hi Lin,
>
> > The load managed by each Bundle is not even. Even if the number of
> partitions managed
>by each bundle is the same, there is no guarantee that the sum of the
> loads of these partitions
>will be the same.
>
> Do we expect that the bundles should have the same loads? The bundle is the
> base unit of the
> load balancer, we can set the high watermark of the bundle, e.g., the
> maximum topics and throughput.
> But the bundle can have different real loads, and if one bundle runs out of
> the high watermark, the bundle
> will be split. Users can tune the high watermark to distribute the loads
> evenly across brokers.
>
> For example, there are 4 bundles with loads 1, 3, 2, 4, the maximum load of
> a bundle is 5 and 2 brokers.
> We can assign bundle 0 and bundle 3 to broker-0 and bundle 1 and bundle 2
> to broker-2.
>
> Of course, this is the ideal situation. If bundle 0 has been assigned to
> broker-0 and bundle 1 has been
> assigned to broker-1. Now, bundle 2 will go to broker 1, and bundle 3 will
> go to broker 1. The loads for each
> broker are 3 and 7. Dynamic programming can help to find an optimized
> solution with more bundle unloads.
>
> So, should we design the bundle to have even loads? It is difficult to
> achieve in reality. And the proposal
> said, "Let each bundle carry the same load as possible". Is it the correct
> direction for the load balancer?
>
> > Doesn't shed loads very well. The existing default policy
> ThresholdShedder has a relatively high usage
>threshold, and various traffic thresholds need to be set. Many clusters
> with high TPS and small message
>bodies may have high CPU but low traffic; And for many small-scale
> clusters, the threshold needs to be
>modified according to the actual business.
>
> Can it be resolved by introducing the entry write/read rate to the bundle
> stats?
>
> > The removed Bundle cannot be well distributed to other Brokers. The load
> information of each Broker
>will be reported at regular intervals, so the judgment of the Leader
> Broker when allocating Bundles cannot
>be guaranteed to be completely correct. Secondly, if there are a large
> number of Bundles to be redistributed,
>the Leader may make the low-load Broker a new high-load node when the
> load information is not up-to-date.
>
> Can we try to force-sync the load data of the brokers before performing the
> distribution of a large number of
> bundles?
>
> For the Goal section in the proposal. It looks like it doesn't map to the
> issues mentioned in the Motivation section.
> IMO, the proposal should clearly describe the Goal, like which problem will
> be resolved with this proposal.
> Both of the above 3 issues or part of them. And what is the high-level
> solution to resolve the issue,
> and what are the pros and cons compared with the existing solution without
> diving into the implementation section.
>
> Another consideration is the default max bundles of a namespace is 128. I
> don't think the common cases that need
> to set 128 partitions for a topic. If the partitions < the bundle's count,
> will the new solution basically be equivalent to
> the current way?
>
> If this is not a general solution for common scenarios. I support making
> the topic-bundle assigner pluggable without
> introducing the implementation to the Pulsar repo. Users can implement
> their own assigner based on the business
> requirement. Pulsar's general solution may not be good for all scenarios,
> but it is better for scalability (bundle split)
> and enough for most common scenarios. We can keep improving the general
> solution for the general requirement
> for the most common scenarios.
>
> Regards,
> Penghui
>
>
> On Wed, Mar 22, 2023 at 9:52 AM Lin Lin  wrote:
>
> >
> > > This appears to be the "round-robin topic-to-bundle mapping" option in
> > > the `fundBundle` function. Is this the only place that needs an update?
> > Can
> > > you list what change is required?
> >
> > In this PIP, we only discuss topic-to-bundle mapping
> > Change is required:
> > 1)
> > When lookup, partitions is assigned to bundle:
> > Lookup -> NamespaceService#getBrokerServiceUrlAsync ->
> > NamespaceService#getBundle

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Nozomi Kurihara]

Hi Baodi,

Thank you for starting the release process.

I think https://github.com/apache/pulsar-client-node/pull/309 seems an
important fix.
Should we include it in the 1.8.2 release or postpone it to the next
release?

Thanks,
Nozomi

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Baodi Shi]

Hi,
I noticed that there is one Reader related issue for this that needs to be
fixed, so I guess it could be included in 1.8.3.

Thanks
Baodi Shi

On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara  wrote:

> Hi Baodi,
>
> Thank you for starting the release process.
>
> I think https://github.com/apache/pulsar-client-node/pull/309 seems an
> important fix.
> Should we include it in the 1.8.2 release or postpone it to the next
> release?
>
> Thanks,
> Nozomi
>
-- 

Thanks
Baodi Shi

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[tison]

Hi Baodi,

> there is one Reader related issue

May you share the link to the issue? There seems no such reference in
PR-309.

Best,
tison.


Baodi Shi  于2023年4月11日周二 16:17写道：

> Hi,
> I noticed that there is one Reader related issue for this that needs to be
> fixed, so I guess it could be included in 1.8.3.
>
> Thanks
> Baodi Shi
>
> On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara  wrote:
>
> > Hi Baodi,
> >
> > Thank you for starting the release process.
> >
> > I think https://github.com/apache/pulsar-client-node/pull/309 seems an
> > important fix.
> > Should we include it in the 1.8.2 release or postpone it to the next
> > release?
> >
> > Thanks,
> > Nozomi
> >
> --
>
> Thanks
> Baodi Shi
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Zike Yang]

Hi,

https://github.com/apache/pulsar-client-node/issues/308 is not a
regression bug. I don't think we need to block this RC.

> May you share the link to the issue? There seems no such reference in PR-309.

I have created an issue to track it:
https://github.com/apache/pulsar-client-node/issues/315

But if you all want to include that fix, I can push a PR soon.

Thanks,
Zike Yang

On Tue, Apr 11, 2023 at 4:29 PM tison  wrote:
>
> Hi Baodi,
>
> > there is one Reader related issue
>
> May you share the link to the issue? There seems no such reference in
> PR-309.
>
> Best,
> tison.
>
>
> Baodi Shi  于2023年4月11日周二 16:17写道：
>
> > Hi,
> > I noticed that there is one Reader related issue for this that needs to be
> > fixed, so I guess it could be included in 1.8.3.
> >
> > Thanks
> > Baodi Shi
> >
> > On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara  wrote:
> >
> > > Hi Baodi,
> > >
> > > Thank you for starting the release process.
> > >
> > > I think https://github.com/apache/pulsar-client-node/pull/309 seems an
> > > important fix.
> > > Should we include it in the 1.8.2 release or postpone it to the next
> > > release?
> > >
> > > Thanks,
> > > Nozomi
> > >
> > --
> >
> > Thanks
> > Baodi Shi
> >

Re: Call for projects and mentors for OSPP 2023

[Yu]

Thanks Dianjing!

This is an activity worth attending based on my previous experience. As a
mentor for OSPP 2021 and 2022, I collaborated with some students and
@urfreespace to create fresh content experiences for Pulsar, such as
Introducing a Bot to Improve the Efficiency of Developing Docs [1] and
Automating Documentation Workflows.

I'll continue to apply to participate as a mentor this year to infuse more
new blood into our community with the project targeting improving content
and website. Will send the application to you later.

[1]
https://docs.google.com/document/d/1bQfZkSu5nG1tNycpmXXtUFn-Z5-h-uqHv6IXsCEySQ8/edit

On Tue, Apr 11, 2023 at 10:53 AM Dianjin Wang  wrote:

> Hi all,
>
> Glad to share that Apache Pulsar is listed at the OSPP 2023 again. This
> year, the Pulsar community can open 7 projects at most.
>
> For OSPP 2023, the project idea will be open from 4/04, 2023 to 04/28,
> 2023(UTC+8). If you have great ideas, please reply to this email by
> following the project template. Then I can help you to submit them.
>
> OSPP asks that Pulsar committers, PMC members, and contributors be the
> mentors; a mentor can only mentor one project. Both mentors and students
> will receive financial awards for completed projects.
>
> You may want to know about OSPP 2022, so refer to this email[1] for
> details.
>
> --
> [Template]
>
> ## Project Info
> Project Name:
> Project Description: (at most 1000 words)
> Difficulty Level:
> - [ ] Basic
> - [ ] Advanced
>
> Project Output Requirements:
> Item 1:__
> Item 2:__
> Item 3:__
> …
>
> Project Technical Requirements:
> Item 1:__
> Item 2:__
> Item 3:__
> …
>
> ## Mentor Info
> Mentor Name:
> Mentor Email:
> --
>
> [1] https://lists.apache.org/thread/7pplcd4c35qjzt2o58qxykkty8qqxvt3
>
> Best,
> Dianjin Wang
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Nozomi Kurihara]

Hi Baodi, Zike,

Thank you for clarifying the issue status.
Although this is not a regression bug, I prefer including the fix to the
current 1.8.2 release since it is easily reproducible and causes unlimited
memory usage increase.

Thanks,
Nozomi

2023年4月11日(火) 18:01 Zike Yang :

> Hi,
>
> https://github.com/apache/pulsar-client-node/issues/308 is not a
> regression bug. I don't think we need to block this RC.
>
> > May you share the link to the issue? There seems no such reference in
> PR-309.
>
> I have created an issue to track it:
> https://github.com/apache/pulsar-client-node/issues/315
>
> But if you all want to include that fix, I can push a PR soon.
>
> Thanks,
> Zike Yang
>
> On Tue, Apr 11, 2023 at 4:29 PM tison  wrote:
> >
> > Hi Baodi,
> >
> > > there is one Reader related issue
> >
> > May you share the link to the issue? There seems no such reference in
> > PR-309.
> >
> > Best,
> > tison.
> >
> >
> > Baodi Shi  于2023年4月11日周二 16:17写道：
> >
> > > Hi,
> > > I noticed that there is one Reader related issue for this that needs
> to be
> > > fixed, so I guess it could be included in 1.8.3.
> > >
> > > Thanks
> > > Baodi Shi
> > >
> > > On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara 
> wrote:
> > >
> > > > Hi Baodi,
> > > >
> > > > Thank you for starting the release process.
> > > >
> > > > I think https://github.com/apache/pulsar-client-node/pull/309 seems
> an
> > > > important fix.
> > > > Should we include it in the 1.8.2 release or postpone it to the next
> > > > release?
> > > >
> > > > Thanks,
> > > > Nozomi
> > > >
> > > --
> > >
> > > Thanks
> > > Baodi Shi
> > >
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Baodi Shi]

Hi, @Nozomi Kurihara 

Ok, We can wait for this issue fixed
https://github.com/apache/pulsar-client-node/issues/315.


Thanks,
Baodi Shi


On Apr 11, 2023 at 17:54:56, Nozomi Kurihara  wrote:

> Hi Baodi, Zike,
>
> Thank you for clarifying the issue status.
> Although this is not a regression bug, I prefer including the fix to the
> current 1.8.2 release since it is easily reproducible and causes unlimited
> memory usage increase.
>
> Thanks,
> Nozomi
>
> 2023年4月11日(火) 18:01 Zike Yang :
>
> Hi,
>
>
> https://github.com/apache/pulsar-client-node/issues/308 is not a
>
> regression bug. I don't think we need to block this RC.
>
>
> > May you share the link to the issue? There seems no such reference in
>
> PR-309.
>
>
> I have created an issue to track it:
>
> https://github.com/apache/pulsar-client-node/issues/315
>
>
> But if you all want to include that fix, I can push a PR soon.
>
>
> Thanks,
>
> Zike Yang
>
>
> On Tue, Apr 11, 2023 at 4:29 PM tison  wrote:
>
> >
>
> > Hi Baodi,
>
> >
>
> > > there is one Reader related issue
>
> >
>
> > May you share the link to the issue? There seems no such reference in
>
> > PR-309.
>
> >
>
> > Best,
>
> > tison.
>
> >
>
> >
>
> > Baodi Shi  于2023年4月11日周二 16:17写道：
>
> >
>
> > > Hi,
>
> > > I noticed that there is one Reader related issue for this that needs
>
> to be
>
> > > fixed, so I guess it could be included in 1.8.3.
>
> > >
>
> > > Thanks
>
> > > Baodi Shi
>
> > >
>
> > > On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara 
>
> wrote:
>
> > >
>
> > > > Hi Baodi,
>
> > > >
>
> > > > Thank you for starting the release process.
>
> > > >
>
> > > > I think https://github.com/apache/pulsar-client-node/pull/309 seems
>
> an
>
> > > > important fix.
>
> > > > Should we include it in the 1.8.2 release or postpone it to the next
>
> > > > release?
>
> > > >
>
> > > > Thanks,
>
> > > > Nozomi
>
> > > >
>
> > > --
>
> > >
>
> > > Thanks
>
> > > Baodi Shi
>
> > >
>
>
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Zike Yang]

Hi, all

I have created a PR
https://github.com/apache/pulsar-client-node/pull/316 for that issue.
PTAL and help review.

Thanks,
Zike Yang

On Tue, Apr 11, 2023 at 5:57 PM Baodi Shi  wrote:
>
> Hi, @Nozomi Kurihara 
>
> Ok, We can wait for this issue fixed
> https://github.com/apache/pulsar-client-node/issues/315.
>
>
> Thanks,
> Baodi Shi
>
>
> On Apr 11, 2023 at 17:54:56, Nozomi Kurihara  wrote:
>
> > Hi Baodi, Zike,
> >
> > Thank you for clarifying the issue status.
> > Although this is not a regression bug, I prefer including the fix to the
> > current 1.8.2 release since it is easily reproducible and causes unlimited
> > memory usage increase.
> >
> > Thanks,
> > Nozomi
> >
> > 2023年4月11日(火) 18:01 Zike Yang :
> >
> > Hi,
> >
> >
> > https://github.com/apache/pulsar-client-node/issues/308 is not a
> >
> > regression bug. I don't think we need to block this RC.
> >
> >
> > > May you share the link to the issue? There seems no such reference in
> >
> > PR-309.
> >
> >
> > I have created an issue to track it:
> >
> > https://github.com/apache/pulsar-client-node/issues/315
> >
> >
> > But if you all want to include that fix, I can push a PR soon.
> >
> >
> > Thanks,
> >
> > Zike Yang
> >
> >
> > On Tue, Apr 11, 2023 at 4:29 PM tison  wrote:
> >
> > >
> >
> > > Hi Baodi,
> >
> > >
> >
> > > > there is one Reader related issue
> >
> > >
> >
> > > May you share the link to the issue? There seems no such reference in
> >
> > > PR-309.
> >
> > >
> >
> > > Best,
> >
> > > tison.
> >
> > >
> >
> > >
> >
> > > Baodi Shi  于2023年4月11日周二 16:17写道：
> >
> > >
> >
> > > > Hi,
> >
> > > > I noticed that there is one Reader related issue for this that needs
> >
> > to be
> >
> > > > fixed, so I guess it could be included in 1.8.3.
> >
> > > >
> >
> > > > Thanks
> >
> > > > Baodi Shi
> >
> > > >
> >
> > > > On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara 
> >
> > wrote:
> >
> > > >
> >
> > > > > Hi Baodi,
> >
> > > > >
> >
> > > > > Thank you for starting the release process.
> >
> > > > >
> >
> > > > > I think https://github.com/apache/pulsar-client-node/pull/309 seems
> >
> > an
> >
> > > > > important fix.
> >
> > > > > Should we include it in the 1.8.2 release or postpone it to the next
> >
> > > > > release?
> >
> > > > >
> >
> > > > > Thanks,
> >
> > > > > Nozomi
> >
> > > > >
> >
> > > > --
> >
> > > >
> >
> > > > Thanks
> >
> > > > Baodi Shi
> >
> > > >
> >
> >
> >

Re: Call for projects and mentors for OSPP 2023

[Xiangying Meng]

Hi Dianjin,

Thank you for sharing the exciting news about Apache Pulsar's participation
in OSPP 2023.
I would also like to express my interest in joining this year's event as a
mentor for the Pulsar project.

We are planning to work on implementing isolation levels for Pulsar.
I will send you the detailed proposal for this project idea soon.

Best regards,
Xiangying

On Tue, Apr 11, 2023 at 5:07 PM Yu  wrote:

> Thanks Dianjing!
>
> This is an activity worth attending based on my previous experience. As a
> mentor for OSPP 2021 and 2022, I collaborated with some students and
> @urfreespace to create fresh content experiences for Pulsar, such as
> Introducing a Bot to Improve the Efficiency of Developing Docs [1] and
> Automating Documentation Workflows.
>
> I'll continue to apply to participate as a mentor this year to infuse more
> new blood into our community with the project targeting improving content
> and website. Will send the application to you later.
>
> [1]
>
> https://docs.google.com/document/d/1bQfZkSu5nG1tNycpmXXtUFn-Z5-h-uqHv6IXsCEySQ8/edit
>
> On Tue, Apr 11, 2023 at 10:53 AM Dianjin Wang 
> wrote:
>
> > Hi all,
> >
> > Glad to share that Apache Pulsar is listed at the OSPP 2023 again. This
> > year, the Pulsar community can open 7 projects at most.
> >
> > For OSPP 2023, the project idea will be open from 4/04, 2023 to 04/28,
> > 2023(UTC+8). If you have great ideas, please reply to this email by
> > following the project template. Then I can help you to submit them.
> >
> > OSPP asks that Pulsar committers, PMC members, and contributors be the
> > mentors; a mentor can only mentor one project. Both mentors and students
> > will receive financial awards for completed projects.
> >
> > You may want to know about OSPP 2022, so refer to this email[1] for
> > details.
> >
> > --
> > [Template]
> >
> > ## Project Info
> > Project Name:
> > Project Description: (at most 1000 words)
> > Difficulty Level:
> > - [ ] Basic
> > - [ ] Advanced
> >
> > Project Output Requirements:
> > Item 1:__
> > Item 2:__
> > Item 3:__
> > …
> >
> > Project Technical Requirements:
> > Item 1:__
> > Item 2:__
> > Item 3:__
> > …
> >
> > ## Mentor Info
> > Mentor Name:
> > Mentor Email:
> > --
> >
> > [1] https://lists.apache.org/thread/7pplcd4c35qjzt2o58qxykkty8qqxvt3
> >
> > Best,
> > Dianjin Wang
> >
>

[VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 2

[Baodi Shi]

Hi everyone,

This is the first release candidate for Apache Pulsar Node.js client,
version 1.8.2.

It fixes the following issues:
https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.2+is%3Aclosed

Please download the source files and review this release candidate:
- Download the source package, verify shasum and asc
- Follow the README.md to build and run the Pulsar Node.js client.

The release candidate package has been published to the npm registry:

https://www.npmjs.com/package/pulsar-client/v/1.8.2-rc.2

You can install it by `npm i pulsar-client@1.8.2-rc.2
--pulsar_binary_host_mirror=https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
and verify the package.

You can refer to this repository to verify tls related features:

https://github.com/shibd/pulsar-client-tls-test

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Source files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.2-rc.2/

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

SHA-512 checksum:

300cc86ab2fe0c433e818aa163f9646f7d23a6d26fd9dd8bcd61d375cfb69e9195eb4b13e37272f9c72a9e426a93f028c516e96546d60b0498f69bc4f9bb7276
 ./apache-pulsar-client-node-1.8.2.tar.gz

The tag to be voted upon:
v1.8.2-rc.2(c5fccfc)
https://github.com/apache/pulsar-client-node/releases/tag/v1.8.2-rc.2

Please review and vote on the release candidate #1 for the version
1.8.2, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Thanks,
Baodi Shi

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 1

[Baodi Shi]

Hi, all

The 1.8.2-rc.2 has been released. Please help verify it.

It contains the following two PRs:

   - https://github.com/apache/pulsar-client-node/pull/316
   - https://github.com/apache/pulsar-client-node/pull/309


https://lists.apache.org/thread/72240tgq5bct78rdslzc689w6h9nb8z0


Thanks,
Baodi Shi


On Apr 11, 2023 at 18:04:39, Zike Yang  wrote:

> Hi, all
>
> I have created a PR
> https://github.com/apache/pulsar-client-node/pull/316 for that issue.
> PTAL and help review.
>
> Thanks,
> Zike Yang
>
> On Tue, Apr 11, 2023 at 5:57 PM Baodi Shi  wrote:
>
>
> Hi, @Nozomi Kurihara 
>
>
> Ok, We can wait for this issue fixed
>
> https://github.com/apache/pulsar-client-node/issues/315.
>
>
>
> Thanks,
>
> Baodi Shi
>
>
>
> On Apr 11, 2023 at 17:54:56, Nozomi Kurihara  wrote:
>
>
> > Hi Baodi, Zike,
>
> >
>
> > Thank you for clarifying the issue status.
>
> > Although this is not a regression bug, I prefer including the fix to the
>
> > current 1.8.2 release since it is easily reproducible and causes
> unlimited
>
> > memory usage increase.
>
> >
>
> > Thanks,
>
> > Nozomi
>
> >
>
> > 2023年4月11日(火) 18:01 Zike Yang :
>
> >
>
> > Hi,
>
> >
>
> >
>
> > https://github.com/apache/pulsar-client-node/issues/308 is not a
>
> >
>
> > regression bug. I don't think we need to block this RC.
>
> >
>
> >
>
> > > May you share the link to the issue? There seems no such reference in
>
> >
>
> > PR-309.
>
> >
>
> >
>
> > I have created an issue to track it:
>
> >
>
> > https://github.com/apache/pulsar-client-node/issues/315
>
> >
>
> >
>
> > But if you all want to include that fix, I can push a PR soon.
>
> >
>
> >
>
> > Thanks,
>
> >
>
> > Zike Yang
>
> >
>
> >
>
> > On Tue, Apr 11, 2023 at 4:29 PM tison  wrote:
>
> >
>
> > >
>
> >
>
> > > Hi Baodi,
>
> >
>
> > >
>
> >
>
> > > > there is one Reader related issue
>
> >
>
> > >
>
> >
>
> > > May you share the link to the issue? There seems no such reference in
>
> >
>
> > > PR-309.
>
> >
>
> > >
>
> >
>
> > > Best,
>
> >
>
> > > tison.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Baodi Shi  于2023年4月11日周二 16:17写道：
>
> >
>
> > >
>
> >
>
> > > > Hi,
>
> >
>
> > > > I noticed that there is one Reader related issue for this that needs
>
> >
>
> > to be
>
> >
>
> > > > fixed, so I guess it could be included in 1.8.3.
>
> >
>
> > > >
>
> >
>
> > > > Thanks
>
> >
>
> > > > Baodi Shi
>
> >
>
> > > >
>
> >
>
> > > > On Tue, Apr 11, 2023 at 15:39 Nozomi Kurihara 
>
> >
>
> > wrote:
>
> >
>
> > > >
>
> >
>
> > > > > Hi Baodi,
>
> >
>
> > > > >
>
> >
>
> > > > > Thank you for starting the release process.
>
> >
>
> > > > >
>
> >
>
> > > > > I think https://github.com/apache/pulsar-client-node/pull/309
> seems
>
> >
>
> > an
>
> >
>
> > > > > important fix.
>
> >
>
> > > > > Should we include it in the 1.8.2 release or postpone it to the
> next
>
> >
>
> > > > > release?
>
> >
>
> > > > >
>
> >
>
> > > > > Thanks,
>
> >
>
> > > > > Nozomi
>
> >
>
> > > > >
>
> >
>
> > > > --
>
> >
>
> > > >
>
> >
>
> > > > Thanks
>
> >
>
> > > > Baodi Shi
>
> >
>
> > > >
>
> >
>
> >
>
> >
>
>

[DISCUSS] File missing on distro and hotfix

[tison]

Hi,

I noticed an issue that the 2.11.0 distro misses a config file for
examples[1]. Since we're closing to the 3.0.0 release process, I'd like to
bring this topic here for:

1. As a reminder for 3.0.0 RMs to check the existence of these files after
making the distro.
2. Discuss if we can re-upload the 2.11.0 bin tarball with a hotfix.

For 2, if we decide to do that, at least we should update the sign file and
checksum file also, and it may go another (lightweight) vote. If we never
mutate released artifacts but only make following patch releases as a fix,
let's conclude explicitly - I'll be glad to know if there is existing
consensus on this topic.

Best,
tison.

[1] https://github.com/apache/pulsar/issues/20063

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 2

[Zili Chen]

Hi Baodi,

When running the tls test you provided, I encountered this error:

$ node tls-oauth2.js

[INFO][ClientConnection:190] [ -> 
pulsar+ssl://baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651]
 Create ClientConnection, timeout=1
[INFO][ConnectionPool:97] Created connection for 
pulsar+ssl://baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
[INFO][ClientConnection:388] [198.18.0.6:63458 -> 198.19.146.80:6651] Connected 
to broker
[ERROR][ClientConnection:488] [198.18.0.6:63458 -> 198.19.146.80:6651] 
Handshake failed: stream truncated
[INFO][ClientConnection:1600] [198.18.0.6:63458 -> 198.19.146.80:6651] 
Connection closed with ConnectError
[ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while 
creating producer on persistent://public/default/test-tls10 -- ConnectError
[INFO][ClientConnection:269] [198.18.0.6:63458 -> 198.19.146.80:6651] Destroyed 
connection
node:internal/process/promises:289
triggerUncaughtException(err, true /* fromPromise */);
^

[Error: Failed to create producer: ConnectError]

Node.js v19.8.1

$ node tls-token.js

[INFO][ClientConnection:190] [ -> 
pulsar+ssl://baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651]
 Create ClientConnection, timeout=1
[INFO][ConnectionPool:97] Created connection for 
pulsar+ssl://baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
[INFO][ClientConnection:388] [198.18.0.6:63459 -> 198.19.146.80:6651] Connected 
to broker
[ERROR][ClientConnection:488] [198.18.0.6:63459 -> 198.19.146.80:6651] 
Handshake failed: stream truncated
[INFO][ClientConnection:1600] [198.18.0.6:63459 -> 198.19.146.80:6651] 
Connection closed with ConnectError
[ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while 
creating producer on persistent://public/default/test-tls10 -- ConnectError
[INFO][ClientConnection:269] [198.18.0.6:63459 -> 198.19.146.80:6651] Destroyed 
connection
node:internal/process/promises:289
triggerUncaughtException(err, true /* fromPromise */);
^

[Error: Failed to create producer: ConnectError]

Node.js v19.8.1

I don't know if it's because the Pulsar URL isn't public or other issues.

Best,
tison.

On 2023/04/11 15:21:37 Baodi Shi wrote:
> Hi everyone,
> 
> This is the first release candidate for Apache Pulsar Node.js client,
> version 1.8.2.
> 
> It fixes the following issues:
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.2+is%3Aclosed
> 
> Please download the source files and review this release candidate:
> - Download the source package, verify shasum and asc
> - Follow the README.md to build and run the Pulsar Node.js client.
> 
> The release candidate package has been published to the npm registry:
> 
> https://www.npmjs.com/package/pulsar-client/v/1.8.2-rc.2
> 
> You can install it by `npm i pulsar-client@1.8.2-rc.2
> --pulsar_binary_host_mirror=https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
> and verify the package.
> 
> You can refer to this repository to verify tls related features:
> 
> https://github.com/shibd/pulsar-client-tls-test
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Source files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.2-rc.2/
> 
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> 
> SHA-512 checksum:
> 
> 300cc86ab2fe0c433e818aa163f9646f7d23a6d26fd9dd8bcd61d375cfb69e9195eb4b13e37272f9c72a9e426a93f028c516e96546d60b0498f69bc4f9bb7276
>  ./apache-pulsar-client-node-1.8.2.tar.gz
> 
> The tag to be voted upon:
> v1.8.2-rc.2(c5fccfc)
> https://github.com/apache/pulsar-client-node/releases/tag/v1.8.2-rc.2
> 
> Please review and vote on the release candidate #1 for the version
> 1.8.2, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Thanks,
> Baodi Shi
> 

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 2

[Baodi Shi]

This example connects test env. Maybe it's network latency or server
instability.

You can try again.

BTW:  The token expires in tls-token.js . I updated it, you can pull the
main branch and try it.

Thanks,
Baodi Shi


On Apr 12, 2023 at 00:22:53, Zili Chen  wrote:

> Hi Baodi,
>
> When running the tls test you provided, I encountered this error:
>
> $ node tls-oauth2.js
>
> [INFO][ClientConnection:190] [ -> pulsar+ssl://
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651]
> Create ClientConnection, timeout=1
> [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> [INFO][ClientConnection:388] [198.18.0.6:63458 -> 198.19.146.80:6651]
> Connected to broker
> [ERROR][ClientConnection:488] [198.18.0.6:63458 -> 198.19.146.80:6651]
> Handshake failed: stream truncated
> [INFO][ClientConnection:1600] [198.18.0.6:63458 -> 198.19.146.80:6651]
> Connection closed with ConnectError
> [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
> creating producer on persistent://public/default/test-tls10 --
> ConnectError
> [INFO][ClientConnection:269] [198.18.0.6:63458 -> 198.19.146.80:6651]
> Destroyed connection
> node:internal/process/promises:289
>triggerUncaughtException(err, true /* fromPromise */);
>^
>
> [Error: Failed to create producer: ConnectError]
>
> Node.js v19.8.1
>
> $ node tls-token.js
>
> [INFO][ClientConnection:190] [ -> pulsar+ssl://
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651]
> Create ClientConnection, timeout=1
> [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> [INFO][ClientConnection:388] [198.18.0.6:63459 -> 198.19.146.80:6651]
> Connected to broker
> [ERROR][ClientConnection:488] [198.18.0.6:63459 -> 198.19.146.80:6651]
> Handshake failed: stream truncated
> [INFO][ClientConnection:1600] [198.18.0.6:63459 -> 198.19.146.80:6651]
> Connection closed with ConnectError
> [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
> creating producer on persistent://public/default/test-tls10 --
> ConnectError
> [INFO][ClientConnection:269] [198.18.0.6:63459 -> 198.19.146.80:6651]
> Destroyed connection
> node:internal/process/promises:289
>triggerUncaughtException(err, true /* fromPromise */);
>^
>
> [Error: Failed to create producer: ConnectError]
>
> Node.js v19.8.1
>
> I don't know if it's because the Pulsar URL isn't public or other issues.
>
> Best,
> tison.
>
> On 2023/04/11 15:21:37 Baodi Shi wrote:
>
> Hi everyone,
>
>
> This is the first release candidate for Apache Pulsar Node.js client,
>
> version 1.8.2.
>
>
> It fixes the following issues:
>
>
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.2+is%3Aclosed
>
>
> Please download the source files and review this release candidate:
>
> - Download the source package, verify shasum and asc
>
> - Follow the README.md to build and run the Pulsar Node.js client.
>
>
> The release candidate package has been published to the npm registry:
>
>
> https://www.npmjs.com/package/pulsar-client/v/1.8.2-rc.2
>
>
> You can install it by `npm i pulsar-client@1.8.2-rc.2
>
> --pulsar_binary_host_mirror=
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>
> and verify the package.
>
>
> You can refer to this repository to verify tls related features:
>
>
> https://github.com/shibd/pulsar-client-tls-test
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
>
> approval, with at least 3 PMC affirmative votes.
>
>
> Source files:
>
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.2-rc.2/
>
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
>
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
>
> SHA-512 checksum:
>
>
>
> 300cc86ab2fe0c433e818aa163f9646f7d23a6d26fd9dd8bcd61d375cfb69e9195eb4b13e37272f9c72a9e426a93f028c516e96546d60b0498f69bc4f9bb7276
>
>  ./apache-pulsar-client-node-1.8.2.tar.gz
>
>
> The tag to be voted upon:
>
> v1.8.2-rc.2(c5fccfc)
>
> https://github.com/apache/pulsar-client-node/releases/tag/v1.8.2-rc.2
>
>
> Please review and vote on the release candidate #1 for the version
>
> 1.8.2, as follows:
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
>
> Thanks,
>
> Baodi Shi
>
>
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 2

[tison]

Thank you! I can successfully run the example.

However, -1 because the source release doesn't contain a NOTICE file.

Other check items including LICENSE, license headers, checksum and GPG sign
should be fine.

Best,
tison.


Baodi Shi  于2023年4月12日周三 09:48写道：

> This example connects test env. Maybe it's network latency or server
> instability.
>
> You can try again.
>
> BTW:  The token expires in tls-token.js . I updated it, you can pull the
> main branch and try it.
>
> Thanks,
> Baodi Shi
>
>
> On Apr 12, 2023 at 00:22:53, Zili Chen  wrote:
>
> > Hi Baodi,
> >
> > When running the tls test you provided, I encountered this error:
> >
> > $ node tls-oauth2.js
> >
> > [INFO][ClientConnection:190] [ -> pulsar+ssl://
> >
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> ]
> > Create ClientConnection, timeout=1
> > [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
> >
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> > [INFO][ClientConnection:388] [198.18.0.6:63458 -> 198.19.146.80:6651]
> > Connected to broker
> > [ERROR][ClientConnection:488] [198.18.0.6:63458 -> 198.19.146.80:6651]
> > Handshake failed: stream truncated
> > [INFO][ClientConnection:1600] [198.18.0.6:63458 -> 198.19.146.80:6651]
> > Connection closed with ConnectError
> > [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
> > creating producer on persistent://public/default/test-tls10 --
> > ConnectError
> > [INFO][ClientConnection:269] [198.18.0.6:63458 -> 198.19.146.80:6651]
> > Destroyed connection
> > node:internal/process/promises:289
> >triggerUncaughtException(err, true /* fromPromise */);
> >^
> >
> > [Error: Failed to create producer: ConnectError]
> >
> > Node.js v19.8.1
> >
> > $ node tls-token.js
> >
> > [INFO][ClientConnection:190] [ -> pulsar+ssl://
> >
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> ]
> > Create ClientConnection, timeout=1
> > [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
> >
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
> > [INFO][ClientConnection:388] [198.18.0.6:63459 -> 198.19.146.80:6651]
> > Connected to broker
> > [ERROR][ClientConnection:488] [198.18.0.6:63459 -> 198.19.146.80:6651]
> > Handshake failed: stream truncated
> > [INFO][ClientConnection:1600] [198.18.0.6:63459 -> 198.19.146.80:6651]
> > Connection closed with ConnectError
> > [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
> > creating producer on persistent://public/default/test-tls10 --
> > ConnectError
> > [INFO][ClientConnection:269] [198.18.0.6:63459 -> 198.19.146.80:6651]
> > Destroyed connection
> > node:internal/process/promises:289
> >triggerUncaughtException(err, true /* fromPromise */);
> >^
> >
> > [Error: Failed to create producer: ConnectError]
> >
> > Node.js v19.8.1
> >
> > I don't know if it's because the Pulsar URL isn't public or other issues.
> >
> > Best,
> > tison.
> >
> > On 2023/04/11 15:21:37 Baodi Shi wrote:
> >
> > Hi everyone,
> >
> >
> > This is the first release candidate for Apache Pulsar Node.js client,
> >
> > version 1.8.2.
> >
> >
> > It fixes the following issues:
> >
> >
> >
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.2+is%3Aclosed
> >
> >
> > Please download the source files and review this release candidate:
> >
> > - Download the source package, verify shasum and asc
> >
> > - Follow the README.md to build and run the Pulsar Node.js client.
> >
> >
> > The release candidate package has been published to the npm registry:
> >
> >
> > https://www.npmjs.com/package/pulsar-client/v/1.8.2-rc.2
> >
> >
> > You can install it by `npm i pulsar-client@1.8.2-rc.2
> >
> > --pulsar_binary_host_mirror=
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
> 
> >
> > and verify the package.
> >
> >
> > You can refer to this repository to verify tls related features:
> >
> >
> > https://github.com/shibd/pulsar-client-tls-test
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> >
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> > Source files:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.2-rc.2/
> >
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> >
> > SHA-512 checksum:
> >
> >
> >
> >
> 300cc86ab2fe0c433e818aa163f9646f7d23a6d26fd9dd8bcd61d375cfb69e9195eb4b13e37272f9c72a9e426a93f028c516e96546d60b0498f69bc4f9bb7276
> >
> >  ./apache-pulsar-client-node-1.8.2.tar.gz
> >
> >
> > The tag to be voted upon:
> >
> > v1.8.2-rc.2(c5fccfc)
> >
> > https://github.com/apache/pulsar-client-node/releases/tag/v1.8.2-rc.2
> >
> >
> > Please re

Re: [VOTE] Pulsar Node.js Client Release 1.8.2 Candidate 2

[Baodi Shi]

Hi, tison.
Thanks for your feedback.

I noticed that the master branch also doesn't have NOTICE files. I'll add
it first and then launch release 1.8.2-rc.3

Thanks,
Baodi Shi


On Apr 12, 2023 at 10:14:20, tison  wrote:

> Thank you! I can successfully run the example.
>
> However, -1 because the source release doesn't contain a NOTICE file.
>
> Other check items including LICENSE, license headers, checksum and GPG sign
> should be fine.
>
> Best,
> tison.
>
>
> Baodi Shi  于2023年4月12日周三 09:48写道：
>
> This example connects test env. Maybe it's network latency or server
>
> instability.
>
>
> You can try again.
>
>
> BTW:  The token expires in tls-token.js . I updated it, you can pull the
>
> main branch and try it.
>
>
> Thanks,
>
> Baodi Shi
>
>
>
> On Apr 12, 2023 at 00:22:53, Zili Chen  wrote:
>
>
> > Hi Baodi,
>
> >
>
> > When running the tls test you provided, I encountered this error:
>
> >
>
> > $ node tls-oauth2.js
>
> >
>
> > [INFO][ClientConnection:190] [ -> pulsar+ssl://
>
> >
>
>
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
>
> ]
>
> > Create ClientConnection, timeout=1
>
> > [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
>
> >
>
>
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
>
> > [INFO][ClientConnection:388] [198.18.0.6:63458 -> 198.19.146.80:6651]
>
> > Connected to broker
>
> > [ERROR][ClientConnection:488] [198.18.0.6:63458 -> 198.19.146.80:6651]
>
> > Handshake failed: stream truncated
>
> > [INFO][ClientConnection:1600] [198.18.0.6:63458 -> 198.19.146.80:6651]
>
> > Connection closed with ConnectError
>
> > [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
>
> > creating producer on persistent://public/default/test-tls10 --
>
> > ConnectError
>
> > [INFO][ClientConnection:269] [198.18.0.6:63458 -> 198.19.146.80:6651]
>
> > Destroyed connection
>
> > node:internal/process/promises:289
>
> >triggerUncaughtException(err, true /* fromPromise */);
>
> >^
>
> >
>
> > [Error: Failed to create producer: ConnectError]
>
> >
>
> > Node.js v19.8.1
>
> >
>
> > $ node tls-token.js
>
> >
>
> > [INFO][ClientConnection:190] [ -> pulsar+ssl://
>
> >
>
>
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
>
> ]
>
> > Create ClientConnection, timeout=1
>
> > [INFO][ConnectionPool:97] Created connection for pulsar+ssl://
>
> >
>
>
> baodi-test-7407d844-b456-49e1-987c-24f1a5c162cb.usce1-whale.test.g.sn2.dev:6651
>
> > [INFO][ClientConnection:388] [198.18.0.6:63459 -> 198.19.146.80:6651]
>
> > Connected to broker
>
> > [ERROR][ClientConnection:488] [198.18.0.6:63459 -> 198.19.146.80:6651]
>
> > Handshake failed: stream truncated
>
> > [INFO][ClientConnection:1600] [198.18.0.6:63459 -> 198.19.146.80:6651]
>
> > Connection closed with ConnectError
>
> > [ERROR][ClientImpl:184] Error Checking/Getting Partition Metadata while
>
> > creating producer on persistent://public/default/test-tls10 --
>
> > ConnectError
>
> > [INFO][ClientConnection:269] [198.18.0.6:63459 -> 198.19.146.80:6651]
>
> > Destroyed connection
>
> > node:internal/process/promises:289
>
> >triggerUncaughtException(err, true /* fromPromise */);
>
> >^
>
> >
>
> > [Error: Failed to create producer: ConnectError]
>
> >
>
> > Node.js v19.8.1
>
> >
>
> > I don't know if it's because the Pulsar URL isn't public or other issues.
>
> >
>
> > Best,
>
> > tison.
>
> >
>
> > On 2023/04/11 15:21:37 Baodi Shi wrote:
>
> >
>
> > Hi everyone,
>
> >
>
> >
>
> > This is the first release candidate for Apache Pulsar Node.js client,
>
> >
>
> > version 1.8.2.
>
> >
>
> >
>
> > It fixes the following issues:
>
> >
>
> >
>
> >
>
>
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.2+is%3Aclosed
>
> >
>
> >
>
> > Please download the source files and review this release candidate:
>
> >
>
> > - Download the source package, verify shasum and asc
>
> >
>
> > - Follow the README.md to build and run the Pulsar Node.js client.
>
> >
>
> >
>
> > The release candidate package has been published to the npm registry:
>
> >
>
> >
>
> > https://www.npmjs.com/package/pulsar-client/v/1.8.2-rc.2
>
> >
>
> >
>
> > You can install it by `npm i pulsar-client@1.8.2-rc.2
>
> >
>
> > --pulsar_binary_host_mirror=
>
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>
> 
>
> >
>
> > and verify the package.
>
> >
>
> >
>
> > You can refer to this repository to verify tls related features:
>
> >
>
> >
>
> > https://github.com/shibd/pulsar-client-tls-test
>
> >
>
> >
>
> > The vote will be open for at least 72 hours. It is adopted by majority
>
> >
>
> > approval, with at least 3 PMC affirmative votes.
>
> >
>
> >
>
> > Source files:
>
> >
>
> >
>
> >
>
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.2-rc.2/
>
> >
>
> >
>
> > Pu

Re: Call for projects and mentors for OSPP 2023

[Xiangying Meng]

## Project Info
Project Name: Pulsar Transaction Isolation Level Enhancement
Project Description: Implement configurable isolation levels for Pulsar
transactions to enhance flexibility. The project will introduce Read
Committed and Read Uncommitted isolation levels and allow consumers to
configure these levels during the building process.
Difficulty Level:
- [x] Basic
- [ ] Advanced

Project Output Requirements:
Item 1: Update the PulsarConsumer builder to accept isolation level
configurations for Read Committed and Read Uncommitted levels.
Item 2: Modify the transaction buffer and dispatching mechanisms to handle
messages based on the chosen isolation level.

Project Technical Requirements:
Item 1: Knowledge of Pulsar transactions, transaction buffer, and
dispatching mechanisms.
Item 2: Proficiency in Java and the Pulsar client API.
Item 3: Understanding of transaction isolation levels and their
implications.


## Mentor Info
Mentor Name:Xiangying Meng
Mentor Email:xiangy...@apache.org

On Tue, Apr 11, 2023 at 6:31 PM Xiangying Meng  wrote:

> Hi Dianjin,
>
> Thank you for sharing the exciting news about Apache Pulsar's
> participation in OSPP 2023.
> I would also like to express my interest in joining this year's event as a
> mentor for the Pulsar project.
>
> We are planning to work on implementing isolation levels for Pulsar.
> I will send you the detailed proposal for this project idea soon.
>
> Best regards,
> Xiangying
>
> On Tue, Apr 11, 2023 at 5:07 PM Yu  wrote:
>
>> Thanks Dianjing!
>>
>> This is an activity worth attending based on my previous experience. As a
>> mentor for OSPP 2021 and 2022, I collaborated with some students and
>> @urfreespace to create fresh content experiences for Pulsar, such as
>> Introducing a Bot to Improve the Efficiency of Developing Docs [1] and
>> Automating Documentation Workflows.
>>
>> I'll continue to apply to participate as a mentor this year to infuse more
>> new blood into our community with the project targeting improving content
>> and website. Will send the application to you later.
>>
>> [1]
>>
>> https://docs.google.com/document/d/1bQfZkSu5nG1tNycpmXXtUFn-Z5-h-uqHv6IXsCEySQ8/edit
>>
>> On Tue, Apr 11, 2023 at 10:53 AM Dianjin Wang 
>> wrote:
>>
>> > Hi all,
>> >
>> > Glad to share that Apache Pulsar is listed at the OSPP 2023 again. This
>> > year, the Pulsar community can open 7 projects at most.
>> >
>> > For OSPP 2023, the project idea will be open from 4/04, 2023 to 04/28,
>> > 2023(UTC+8). If you have great ideas, please reply to this email by
>> > following the project template. Then I can help you to submit them.
>> >
>> > OSPP asks that Pulsar committers, PMC members, and contributors be the
>> > mentors; a mentor can only mentor one project. Both mentors and students
>> > will receive financial awards for completed projects.
>> >
>> > You may want to know about OSPP 2022, so refer to this email[1] for
>> > details.
>> >
>> > --
>> > [Template]
>> >
>> > ## Project Info
>> > Project Name:
>> > Project Description: (at most 1000 words)
>> > Difficulty Level:
>> > - [ ] Basic
>> > - [ ] Advanced
>> >
>> > Project Output Requirements:
>> > Item 1:__
>> > Item 2:__
>> > Item 3:__
>> > …
>> >
>> > Project Technical Requirements:
>> > Item 1:__
>> > Item 2:__
>> > Item 3:__
>> > …
>> >
>> > ## Mentor Info
>> > Mentor Name:
>> > Mentor Email:
>> > --
>> >
>> > [1] https://lists.apache.org/thread/7pplcd4c35qjzt2o58qxykkty8qqxvt3
>> >
>> > Best,
>> > Dianjin Wang
>> >
>>
>

Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin

[Lin Lin]

As I mentioned in the implementation of PIP, we will plug-in the partition 
assignment strategy. 

However, in the same cluster, it is impossible for some Brokers to use 
consistent hashing and some Brokers to use round robin.

On 2023/04/11 07:37:19 Xiangying Meng wrote:
> Hi Linlin,
> > This is an incompatible modification, so the entire cluster needs to be
> upgraded, not just a part of the nodes
> 
> Appreciate your contribution to the new feature in PIP-255.
>  I have a question regarding the load-balancing aspect of this feature.
> 
> You mentioned that this is an incompatible modification,
> and the entire cluster needs to be upgraded, not just a part of the nodes.
>  I was wondering why we can only have one load-balancing strategy.
> Would it be possible to abstract the logic here and make it an optional
> choice?
> This way, we could have multiple load-balancing strategies,
> such as hash-based, round-robin, etc., available for users to choose from.
> 
> I'd love to hear your thoughts on this.
> 
> Best regards,
> Xiangying
> 
> On Mon, Apr 10, 2023 at 8:23 PM PengHui Li  wrote:
> 
> > Hi Lin,
> >
> > > The load managed by each Bundle is not even. Even if the number of
> > partitions managed
> >by each bundle is the same, there is no guarantee that the sum of the
> > loads of these partitions
> >will be the same.
> >
> > Do we expect that the bundles should have the same loads? The bundle is the
> > base unit of the
> > load balancer, we can set the high watermark of the bundle, e.g., the
> > maximum topics and throughput.
> > But the bundle can have different real loads, and if one bundle runs out of
> > the high watermark, the bundle
> > will be split. Users can tune the high watermark to distribute the loads
> > evenly across brokers.
> >
> > For example, there are 4 bundles with loads 1, 3, 2, 4, the maximum load of
> > a bundle is 5 and 2 brokers.
> > We can assign bundle 0 and bundle 3 to broker-0 and bundle 1 and bundle 2
> > to broker-2.
> >
> > Of course, this is the ideal situation. If bundle 0 has been assigned to
> > broker-0 and bundle 1 has been
> > assigned to broker-1. Now, bundle 2 will go to broker 1, and bundle 3 will
> > go to broker 1. The loads for each
> > broker are 3 and 7. Dynamic programming can help to find an optimized
> > solution with more bundle unloads.
> >
> > So, should we design the bundle to have even loads? It is difficult to
> > achieve in reality. And the proposal
> > said, "Let each bundle carry the same load as possible". Is it the correct
> > direction for the load balancer?
> >
> > > Doesn't shed loads very well. The existing default policy
> > ThresholdShedder has a relatively high usage
> >threshold, and various traffic thresholds need to be set. Many clusters
> > with high TPS and small message
> >bodies may have high CPU but low traffic; And for many small-scale
> > clusters, the threshold needs to be
> >modified according to the actual business.
> >
> > Can it be resolved by introducing the entry write/read rate to the bundle
> > stats?
> >
> > > The removed Bundle cannot be well distributed to other Brokers. The load
> > information of each Broker
> >will be reported at regular intervals, so the judgment of the Leader
> > Broker when allocating Bundles cannot
> >be guaranteed to be completely correct. Secondly, if there are a large
> > number of Bundles to be redistributed,
> >the Leader may make the low-load Broker a new high-load node when the
> > load information is not up-to-date.
> >
> > Can we try to force-sync the load data of the brokers before performing the
> > distribution of a large number of
> > bundles?
> >
> > For the Goal section in the proposal. It looks like it doesn't map to the
> > issues mentioned in the Motivation section.
> > IMO, the proposal should clearly describe the Goal, like which problem will
> > be resolved with this proposal.
> > Both of the above 3 issues or part of them. And what is the high-level
> > solution to resolve the issue,
> > and what are the pros and cons compared with the existing solution without
> > diving into the implementation section.
> >
> > Another consideration is the default max bundles of a namespace is 128. I
> > don't think the common cases that need
> > to set 128 partitions for a topic. If the partitions < the bundle's count,
> > will the new solution basically be equivalent to
> > the current way?
> >
> > If this is not a general solution for common scenarios. I support making
> > the topic-bundle assigner pluggable without
> > introducing the implementation to the Pulsar repo. Users can implement
> > their own assigner based on the business
> > requirement. Pulsar's general solution may not be good for all scenarios,
> > but it is better for scalability (bundle split)
> > and enough for most common scenarios. We can keep improving the general
> > solution for the general requirement
> > for the most common scenarios.
> >
> > Regards,
> > Penghui
> >
> >

[NOTICE] Enter code freeze phase of Pulsar 3.0.0 release

[Zike Yang]

Hi, all

We would like to inform you that the Pulsar 3.0.0 release has entered
the code freeze phase, with the branch-3.0 cut and RC1 currently being
built.

As per our release policy [0]. from the code-freeze point, to minimize
the risk of delaying the release, only bug fixes involving a
regression of behavior compared to a previous release should be
allowed. Occasional exceptions will be possible after higher scrutiny
of the change.

If you have any changes that need to be included in this release,
please leave a comment under this email thread. Please note that don't
cherry-pick any changes to the branch-3.0 without discussion and
consensus under this email thread.

We have created a channel `pulsar-release-3_0` [1] to coordinate the
release work. If you want to check the progress of this release, you
can refer to this channel. But please note that this is only for
coordination but not for the cherry-pick discussion.

If you have any questions, please feel free to leave any comments here.

[0] https://pulsar.apache.org/contribute/release-policy/#release-cycles
[1] https://apache-pulsar.slack.com/archives/C052JAH3W7N

Thanks,
Zike Yang