[DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Jiuming Tao]

Hi,

I’ve opened a new PIP to discuss:

Currently, in Pulsar, we have a configuration named 
`subscriptionExpirationTimeMinutes` to manage the subscription expiration of 
`PersistentTopic` and `NonPersistentTopic`.

When we set a value which is greater than 0 to 
`subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic` and 
`NonPersistentTopic`. Their inactive subscriptions will get expired and will 
clean automatically.

For `NonPersistentTopic`, its subscriptions can be clean because we don't 
guarantee its data integrity. But for `PersistentTopic`, if we clean its 
subscriptions automatically may lead to data loss.

However, their subscription expiration is managed by the same 
configuration(`subscriptionExpirationTimeMinutes`), we can't manage their 
subscription expiration independently.

So I want to introduce a new configuration named 
`nonPersistentSubscriptionExpirationTimeMinutes` to manage 
`NonPersistentTopic`'s subscription expiration.


Please feel free to leave your comments, the PIP link: 
https://github.com/apache/pulsar/issues/19448 



Thanks,
Tao Jiuming

[Discuss] Review the new structure of client library documentation

[Jun Ma]

Hi everyone,

To improve the developer experience and onboard Pulsar users more efficiently, 
I proposed PIP-228 to refactor the client library documentation [1] and got 
three binding votes [2].

@RobertIndie and I have been working on implementing the information 
architecture changes through this pull request [3] by mapping the existing docs 
to users’ learning paths. Note that this PR only focuses on the structure 
improvements, while the detailed content itself definitely needs to be reviewed 
and enriched through a series of pull requests later.

We don’t want to throw any surprises without staging it, so this is a call to 
invite more eyes to preview the new structure and make sure everyone who cares 
about these topics is informed of this upcoming change.

Check out the new structure and add your comments:)


[1]: https://github.com/apache/pulsar/issues/18822
[2]: https://lists.apache.org/thread/5g0mhvks90nx4sh7fnpmkqj6k2dvxjpp
[3]: https://github.com/apache/pulsar-site/pull/393


Best,
Jun

[DISCUSS] Fix OAuth2 regression for C++, Python, Node.js clients

[Yunze Xu]

Hi all,

Recently I found a serious bug for OAuth2 authentication [1]. Here are
some related issues: [2], [3]. The affected client versions are:
- C++ client 3.0.0, 3.1.0, 3.1.1
- Python client 3.0.0
- Node.js client 1.8.0

Users have to downgrade back to old releases, which might suffer the
CVE-2022-33684 [4]. Another workaround is described in [1] but it's
very hacky. I've opened a PR [5] to fix it.

After it's merged, we should release:
- C++ client 3.1.2
- Python client 3.1.0 (VOTE for candidate 2 is passed, but I want to
delay it to wait C++ client 3.1.2)
- Node.js client 1.8.1

[1] https://github.com/apache/pulsar-client-cpp/issues/184
[2] https://github.com/apache/pulsar-client-node/issues/281
[3] https://github.com/apache/pulsar-client-python/issues/88
[4] https://lists.apache.org/thread/ky1ssskvkj00y36k7nys9b5gm5jjrzwv
[5] https://github.com/apache/pulsar-client-cpp/pull/190

Thanks,
Yunze

[ANNOUNCE] Released Reactive client for Apache Pulsar, version 0.2.0

[Christophe Bornet]

The Apache Pulsar team is proud to announce the Reactive client for Apache
Pulsar, version 0.2.0.

The Reactive client for Apache Pulsar can be used together with any
Reactive Streams implementation on the JVM.
Examples include Project Reactor / Spring Reactive, Akka Streams, RxJava 3,
Vert.x, SmallRye Mutiny (RedHat/Quarkus), ServiceTalk and others (Reactive
Streams to JDK Flow adapters might be needed in some cases.)

Reactive programming is about non-blocking applications that are
asynchronous and event-driven and require a small number of threads to
scale.
The Reactive client for Apache Pulsar supports non-blocking, reactive &
asynchronous back pressure for producing and consuming messages.
The library provides an interface module "pulsar-client-reactive-api" that
abstracts the Reactive client API.
This interface is currently implemented by wrapping the Apache Pulsar Java
client and adapting the existing asynchronous Java API to the Reactive
client API.
The decoupled reactive interfaces makes it possible to implement a fully
native reactive client in the future.

Source package is available for download at
https://downloads.apache.org/pulsar/pulsar-client-reactive-0.2.0/

Maven packages are available in Maven central.
Please check https://github.com/apache/pulsar-client-reactive#getting-it
for usage with Maven or Gradle builds.

Release Notes are at:
https://github.com/apache/pulsar-client-reactive/releases/tag/v0.2.0

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[PengHui Li]

I think we are talking about the durable subscription of the non-persistent
topic, right? For the Reader API (non-durable subscription), the
subscription will be
removed after the reader disconnects. If not, it should be BUG.

IMO, we should remove the subscription automatically if all the consumers
are disconnected for the non-persistent topic. The consumers can only get
the new incoming messages after they connect to the non-persistent topic.
It looks like even if we leave the subscription there; it will
help nothing.
The behavior should be the same as creating a new subscription with the
same subscription name.

For subscription management, users are not able to disable the subscription
auto-creation of the non-persistent topic? We don't have any metadata
persist
in the metadata store. After the broker restarts, we will lose everything.

So I think we don't need to introduce a new configuration. Instead, we
should
figure out if there is a strong reason to leave the durable subscription
after
all the consumers are disconnected. If there is no strong reason, I tend to
fix
incorrect behavior directly.

Best,
Penghui




On Tue, Feb 7, 2023 at 4:14 PM Jiuming Tao 
wrote:

> Hi,
>
> I’ve opened a new PIP to discuss:
>
> Currently, in Pulsar, we have a configuration named
> `subscriptionExpirationTimeMinutes` to manage the subscription expiration
> of `PersistentTopic` and `NonPersistentTopic`.
>
> When we set a value which is greater than 0 to
> `subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic`
> and `NonPersistentTopic`. Their inactive subscriptions will get expired and
> will clean automatically.
>
> For `NonPersistentTopic`, its subscriptions can be clean because we don't
> guarantee its data integrity. But for `PersistentTopic`, if we clean its
> subscriptions automatically may lead to data loss.
>
> However, their subscription expiration is managed by the same
> configuration(`subscriptionExpirationTimeMinutes`), we can't manage their
> subscription expiration independently.
>
> So I want to introduce a new configuration named
> `nonPersistentSubscriptionExpirationTimeMinutes` to manage
> `NonPersistentTopic`'s subscription expiration.
>
>
> Please feel free to leave your comments, the PIP link:
> https://github.com/apache/pulsar/issues/19448 <
> https://github.com/apache/pulsar/issues/19448>
>
>
> Thanks,
> Tao Jiuming

Re: [VOTE] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Rui Fu]

+1

Best,

Rui Fu
On Jan 30, 2023 at 19:12 +0800, Alexander Preuss 
, wrote:
> Now that the DISCUSSION thread has received some more input and some time
> has passed I would like to start the VOTE again.
>
> Best,
> Alexander
>
> On Tue, Jan 17, 2023 at 3:21 AM Bonan Hou 
> wrote:
>
> > +1
> >
> > Bonan Hou
> >
> > On Mon, Jan 16, 2023 at 9:56 PM Alexander Preuss
> >  wrote:
> >
> > > Hi everyone,
> > >
> > > I'm starting the VOTE for PIP-237: Make PulsarAdmin accessible in
> > > SinkContext and SourceContext:
> > > https://github.com/apache/pulsar/issues/19123
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/b8r6b4p87qzfgfwmqfql7mqx8kmgorxg
> > >
> > > I'd like to go ahead with implementing option B presented in the PIP
> > > description on GitHub.
> > >
> > > Thanks,
> > > Alexander
> > >
> >

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Enrico Olivelli]

Penghui,

Il giorno mar 7 feb 2023 alle ore 12:58 PengHui Li
 ha scritto:
>
> I think we are talking about the durable subscription of the non-persistent
> topic, right? For the Reader API (non-durable subscription), the
> subscription will be
> removed after the reader disconnects. If not, it should be BUG.
>
> IMO, we should remove the subscription automatically if all the consumers
> are disconnected for the non-persistent topic. The consumers can only get
> the new incoming messages after they connect to the non-persistent topic.
> It looks like even if we leave the subscription there; it will
> help nothing.
> The behavior should be the same as creating a new subscription with the
> same subscription name.
>
> For subscription management, users are not able to disable the subscription
> auto-creation of the non-persistent topic? We don't have any metadata
> persist
> in the metadata store. After the broker restarts, we will lose everything.
>
> So I think we don't need to introduce a new configuration. Instead, we
> should
> figure out if there is a strong reason to leave the durable subscription
> after
> all the consumers are disconnected. If there is no strong reason, I tend to
> fix
> incorrect behavior directly.

I agree
let's go deep into the real problem

Enrico

>
> Best,
> Penghui
>
>
>
>
> On Tue, Feb 7, 2023 at 4:14 PM Jiuming Tao 
> wrote:
>
> > Hi,
> >
> > I’ve opened a new PIP to discuss:
> >
> > Currently, in Pulsar, we have a configuration named
> > `subscriptionExpirationTimeMinutes` to manage the subscription expiration
> > of `PersistentTopic` and `NonPersistentTopic`.
> >
> > When we set a value which is greater than 0 to
> > `subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic`
> > and `NonPersistentTopic`. Their inactive subscriptions will get expired and
> > will clean automatically.
> >
> > For `NonPersistentTopic`, its subscriptions can be clean because we don't
> > guarantee its data integrity. But for `PersistentTopic`, if we clean its
> > subscriptions automatically may lead to data loss.
> >
> > However, their subscription expiration is managed by the same
> > configuration(`subscriptionExpirationTimeMinutes`), we can't manage their
> > subscription expiration independently.
> >
> > So I want to introduce a new configuration named
> > `nonPersistentSubscriptionExpirationTimeMinutes` to manage
> > `NonPersistentTopic`'s subscription expiration.
> >
> >
> > Please feel free to leave your comments, the PIP link:
> > https://github.com/apache/pulsar/issues/19448 <
> > https://github.com/apache/pulsar/issues/19448>
> >
> >
> > Thanks,
> > Tao Jiuming

Re: [ANNOUNCE] Released Reactive client for Apache Pulsar, version 0.2.0

[Enrico Olivelli]

Congratulations !

This is another great step forward for the Pulsar community.

Looking forward for the official Spring Integration that will leverage
this client

Enrico

Il giorno mar 7 feb 2023 alle ore 12:04 Christophe Bornet
 ha scritto:
>
> The Apache Pulsar team is proud to announce the Reactive client for Apache 
> Pulsar, version 0.2.0.
>
> The Reactive client for Apache Pulsar can be used together with any Reactive 
> Streams implementation on the JVM.
> Examples include Project Reactor / Spring Reactive, Akka Streams, RxJava 3, 
> Vert.x, SmallRye Mutiny (RedHat/Quarkus), ServiceTalk and others (Reactive 
> Streams to JDK Flow adapters might be needed in some cases.)
>
> Reactive programming is about non-blocking applications that are asynchronous 
> and event-driven and require a small number of threads to scale.
> The Reactive client for Apache Pulsar supports non-blocking, reactive & 
> asynchronous back pressure for producing and consuming messages.
> The library provides an interface module "pulsar-client-reactive-api" that 
> abstracts the Reactive client API.
> This interface is currently implemented by wrapping the Apache Pulsar Java 
> client and adapting the existing asynchronous Java API to the Reactive client 
> API.
> The decoupled reactive interfaces makes it possible to implement a fully 
> native reactive client in the future.
>
> Source package is available for download at 
> https://downloads.apache.org/pulsar/pulsar-client-reactive-0.2.0/
>
> Maven packages are available in Maven central.
> Please check https://github.com/apache/pulsar-client-reactive#getting-it for 
> usage with Maven or Gradle builds.
>
> Release Notes are at:
> https://github.com/apache/pulsar-client-reactive/releases/tag/v0.2.0
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The Pulsar Team

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Jiuming Tao]

> I think we are talking about the durable subscription of the non-persistent
> topic, right?

Yes, non-durable subscriptions created by Reader, if the reader disconnected, 
the subscription will be deleted automatically.


For `NonPersistentDispatcher`, it has the following implementations:
1. NonPersistentDispatcherMultipleConsumers
2. NonPersistentDispatcherSingleActiveConsumer
3. NonPersistentStickyKeyDispatcherMultipleConsumers

For `NonPersistentDispatcherSingleActiveConsumer`, auto delete the subscription 
when there isn’t consumer connected affect nothing.

But for `NonPersistentDispatcherMultipleConsumers`, 
`NonPersistentStickyKeyDispatcherMultipleConsumers`, I’m not sure that if it 
matters when  we auto delete the subscriptions which has no consumers connected.

If it doesn’t matter, I agree with you. 

Auto delete the durable subscriptions when there aren’t consumers connected to 
the non-persistent topic.


Thanks,
Tao Jiuming


> 2023年2月7日 19:58，PengHui Li  写道：
> 
> I think we are talking about the durable subscription of the non-persistent
> topic, right? For the Reader API (non-durable subscription), the
> subscription will be
> removed after the reader disconnects. If not, it should be BUG.
> 
> IMO, we should remove the subscription automatically if all the consumers
> are disconnected for the non-persistent topic. The consumers can only get
> the new incoming messages after they connect to the non-persistent topic.
> It looks like even if we leave the subscription there; it will
> help nothing.
> The behavior should be the same as creating a new subscription with the
> same subscription name.
> 
> For subscription management, users are not able to disable the subscription
> auto-creation of the non-persistent topic? We don't have any metadata
> persist
> in the metadata store. After the broker restarts, we will lose everything.
> 
> So I think we don't need to introduce a new configuration. Instead, we
> should
> figure out if there is a strong reason to leave the durable subscription
> after
> all the consumers are disconnected. If there is no strong reason, I tend to
> fix
> incorrect behavior directly.
> 
> Best,
> Penghui
> 
> 
> 
> 
> On Tue, Feb 7, 2023 at 4:14 PM Jiuming Tao  >
> wrote:
> 
>> Hi,
>> 
>> I’ve opened a new PIP to discuss:
>> 
>> Currently, in Pulsar, we have a configuration named
>> `subscriptionExpirationTimeMinutes` to manage the subscription expiration
>> of `PersistentTopic` and `NonPersistentTopic`.
>> 
>> When we set a value which is greater than 0 to
>> `subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic`
>> and `NonPersistentTopic`. Their inactive subscriptions will get expired and
>> will clean automatically.
>> 
>> For `NonPersistentTopic`, its subscriptions can be clean because we don't
>> guarantee its data integrity. But for `PersistentTopic`, if we clean its
>> subscriptions automatically may lead to data loss.
>> 
>> However, their subscription expiration is managed by the same
>> configuration(`subscriptionExpirationTimeMinutes`), we can't manage their
>> subscription expiration independently.
>> 
>> So I want to introduce a new configuration named
>> `nonPersistentSubscriptionExpirationTimeMinutes` to manage
>> `NonPersistentTopic`'s subscription expiration.
>> 
>> 
>> Please feel free to leave your comments, the PIP link:
>> https://github.com/apache/pulsar/issues/19448 <
>> https://github.com/apache/pulsar/issues/19448 
>> >
>> 
>> 
>> Thanks,
>> Tao Jiuming

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Jiuming Tao]

> I think we are talking about the durable subscription of the non-persistent
> topic, right?

Yes, non-durable subscriptions created by Reader, if the reader disconnected, 
the subscription will be deleted automatically.


For `NonPersistentDispatcher`, it has the following implementations:
1. NonPersistentDispatcherMultipleConsumers
2. NonPersistentDispatcherSingleActiveConsumer
3. NonPersistentStickyKeyDispatcherMultipleConsumers

For `NonPersistentDispatcherSingleActiveConsumer`, auto delete the subscription 
when there isn’t consumer connected affect nothing.

But for `NonPersistentDispatcherMultipleConsumers`, 
`NonPersistentStickyKeyDispatcherMultipleConsumers`, I’m not sure that if it 
matters when  we auto delete the subscriptions which has no consumers connected.

If it doesn’t matter, I agree with you. 

Auto delete the durable subscriptions when there aren’t consumers connected to 
the non-persistent topic.


Thanks,
Tao Jiuming


> 2023年2月7日 19:58，PengHui Li mailto:peng...@apache.org>> 
> 写道：
> 
> I think we are talking about the durable subscription of the non-persistent
> topic, right? For the Reader API (non-durable subscription), the
> subscription will be
> removed after the reader disconnects. If not, it should be BUG.
> 
> IMO, we should remove the subscription automatically if all the consumers
> are disconnected for the non-persistent topic. The consumers can only get
> the new incoming messages after they connect to the non-persistent topic.
> It looks like even if we leave the subscription there; it will
> help nothing.
> The behavior should be the same as creating a new subscription with the
> same subscription name.
> 
> For subscription management, users are not able to disable the subscription
> auto-creation of the non-persistent topic? We don't have any metadata
> persist
> in the metadata store. After the broker restarts, we will lose everything.
> 
> So I think we don't need to introduce a new configuration. Instead, we
> should
> figure out if there is a strong reason to leave the durable subscription
> after
> all the consumers are disconnected. If there is no strong reason, I tend to
> fix
> incorrect behavior directly.
> 
> Best,
> Penghui
> 
> 
> 
> 
> On Tue, Feb 7, 2023 at 4:14 PM Jiuming Tao  >
> wrote:
> 
>> Hi,
>> 
>> I’ve opened a new PIP to discuss:
>> 
>> Currently, in Pulsar, we have a configuration named
>> `subscriptionExpirationTimeMinutes` to manage the subscription expiration
>> of `PersistentTopic` and `NonPersistentTopic`.
>> 
>> When we set a value which is greater than 0 to
>> `subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic`
>> and `NonPersistentTopic`. Their inactive subscriptions will get expired and
>> will clean automatically.
>> 
>> For `NonPersistentTopic`, its subscriptions can be clean because we don't
>> guarantee its data integrity. But for `PersistentTopic`, if we clean its
>> subscriptions automatically may lead to data loss.
>> 
>> However, their subscription expiration is managed by the same
>> configuration(`subscriptionExpirationTimeMinutes`), we can't manage their
>> subscription expiration independently.
>> 
>> So I want to introduce a new configuration named
>> `nonPersistentSubscriptionExpirationTimeMinutes` to manage
>> `NonPersistentTopic`'s subscription expiration.
>> 
>> 
>> Please feel free to leave your comments, the PIP link:
>> https://github.com/apache/pulsar/issues/19448 
>>  <
>> https://github.com/apache/pulsar/issues/19448 
>> >
>> 
>> 
>> Thanks,
>> Tao Jiuming

Pulsar Flaky test report 2023-02-07 for PR builds in CI

[Nicolò Boschi]

Dear community,
Here's a report of the flaky tests in Pulsar CI during the observation
period of 2023-01-31 - 2023-02-07

https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-01-31-to-2023-02-07

More flaky test issues:
https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue


To coordinate the work,
1) please search for an existing issues or search for all flaky issues with
"flaky" or the test class name (without package) in the search:
https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
2) If there isn't an issue for a particular flaky test failure that you'd
like to fix, please create an issue using the "Flaky test" template at
https://github.com/apache/pulsar/issues/new/choose
3) Please comment on the issue that you are working on.

I'm looking forward to more contributors joining the efforts. Please join
the #testing channel on Slack if you'd like to ask questions and tips about
reproducing flaky tests locally and how to fix them.
Sharing stories about fixing flaky tests is also helpful for sharing the
knowledge about how flaky tests can be fixed. That's also a valuable way to
contribute.
Some flaky tests might be actual real production code bugs. Fixing
the flaky test might result in fixing a real production code bug.

More contributions are welcome! Please keep up the good work!

P.S.
The flaky test reporter was originally coded by Lari. Lately, we simplified
the way to generate the report to allow more people to access the report
and for scheduling purpose.
There are instructions on how to generate it here:
https://github.com/nicoloboschi/pulsar-flakes#generate-a-new-report


Thanks,
Nicolò Boschi

Re: [DISCUSS] Redundant ServiceUrlProvider design and improper use of PIP-121

[Asaf Mesika]

Oh, thanks Baodi - now I understand what he meant by "calling rebuild".
I agree, it's not very intuitive.

Maybe?

client.update(ConnectionInfo.builder()
  .setServiceUrl()
  .set...()
  .build())


On Mon, Feb 6, 2023 at 2:39 AM Baodi Shi  wrote:

>  I thought about it for a moment, and this is not a good idea.
>
> Using newBuild may be understood by the user as recreating the client
> object. It is still intuitive to add the `updateXXX` method.
>
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年2月5日 23:01:57 上，Asaf Mesika  写道：
>
> > Can you please write rebuild example?
> >
> > On Sun, Feb 5, 2023 at 4:05 PM Baodi Shi  wrote:
> >
> > >
> >
> > > Not completely different IMO. When we want to update the service URL,
> >
> > >
> >
> > > we usually want to connect to another cluster. However, the cluster
> >
> > >
> >
> > > could have configured authentication, TLS for security, but we cannot
> >
> > >
> >
> > > guarantee two clusters share the same TLS certificates or
> >
> > >
> >
> > > authentication configs. I think we should group these necessary
> >
> > >
> >
> > > information into a common class like `ConnectInfo`. Then, just expose
> >
> > >
> >
> > > a method to update the `ConnectInfo` so that code could be more
> >
> > >
> >
> > > scalable.
> >
> > >
> >
> > >
> >
> > It makes sense to me.
> >
> >
> > Alternatively, can we directly support rebuilding PulsarClient?
> >
> >
> > PulsarClient client = PulsarClient.builder()
> >
> > .serviceUrl("old url")
> >
> > .tlsTrustCertsFilePath()
> >
> > .statsInterval(1, TimeUnit.SECONDS)
> >
> > .build();
> >
> >
> > client.newBuilder()
> >
> > .serviceUrl("new url")
> >
> > // tls relate
> >
> > .tlsTrustCertsFilePath()
> >
> > .statsInterval(2, TimeUnit.SECONDS)
> >
> > .build();
> >
> >
> >
> > When calling rebuild, the new configuration overwrites the old
> >
> > configuration, and if there are no settings, the old configuration is
> used.
> >
> >
> > 在 2023年2月2日 23:02:21 上，Asaf Mesika  写道：
> >
> >
> > > Ok, makes sense.
> >
> > >
> >
> > >
> >
> > > On Thu, Feb 2, 2023 at 4:02 PM Yunze Xu 
> >
> > > wrote:
> >
> > >
> >
> > > Not completely different IMO. When we want to update the service URL,
> >
> > >
> >
> > > we usually want to connect to another cluster. However, the cluster
> >
> > >
> >
> > > could have configured authentication, TLS for security, but we cannot
> >
> > >
> >
> > > guarantee two clusters share the same TLS certificates or
> >
> > >
> >
> > > authentication configs. I think we should group these necessary
> >
> > >
> >
> > > information into a common class like `ConnectInfo`. Then, just expose
> >
> > >
> >
> > > a method to update the `ConnectInfo` so that code could be more
> >
> > >
> >
> > > scalable.
> >
> > >
> >
> > >
> >
> > > Thanks,
> >
> > >
> >
> > > Yunze
> >
> > >
> >
> > >
> >
> > > On Thu, Feb 2, 2023 at 9:39 PM Asaf Mesika 
> >
> > wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Ok, the last two methods IMO are a completely different topic.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > I wonder what the maintainers think about this.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Jan 30, 2023 at 4:16 AM Yunze Xu
>  >
> > >
> >
> > >
> >
> > > > wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > In summary, the core issue is that ServiceUrlProvider could be
> >
> > >
> >
> > > > > misused. A user might want to implement its own ServiceUrlProvider
> by
> >
> > >
> >
> > > > > changing the returned value of `get` method but it actually does
> not
> >
> > >
> >
> > > > > work. The built-in implementations like AutoFailover actually work
> by
> >
> > >
> >
> > > > > these `updateXXX` methods. So it's meaningless to provide them as
> >
> > >
> >
> > > > > ServiceUrlProvider implementations.
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > To answer your question directly: totally yes. But not only
> >
> > >
> >
> > > > > `setServiceUrl` (the name is `updateServiceUrl` in code),  the
> >
> > >
> >
> > > > > `updateAuthentication` and `updateTlsTrustCertsFilePath` methods,
> >
> > >
> >
> > > > > which are not exposed to `PulsarClient`, should also be exposed.
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > Thanks,
> >
> > >
> >
> > > > > Yunze
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > On Sun, Jan 29, 2023 at 11:57 PM Asaf Mesika <
> asaf.mes...@gmail.com>
> >
> > >
> >
> > > > > wrote:
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > > > > Yunze - in summary - your proposal is to get rid of the
> >
> > >
> >
> > > > > ServiceUrlProvider
> >
> > >
> >
> > > > > > right? You just want to have setServiceUrl on the client instead?
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > > > > On Wed, Jan 25, 2023 at 2:03 PM Yunze Xu
> >
> > >
> >
> > > 
> >
> > >
> >
> > > > > > wrote:

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

[Heesung Sohn]

On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao  wrote:

> On 2023/02/03 18:14:53 Heesung Sohn wrote:
> > There are some cases to trigger it.
> > 1. A cursor be removed.
> > 2. Close the current ledger and create a new ledger.
> > 3. Consumer ack the message, the slowest cursor move forward.
> > 4. User trigger truncateTopic by admin.
> >
> > I see that this pip is for the internal ledger deletion cases(1-3 above),
> > and it appears that such internal deletion operations do not require
> > pre-validation or status checks(and no additional iops on the metadata
> > store). I agree that we need a separate pip for async admin operations.
>
> This pip is also applied to case 4. All cases will invoke
> `trimConsumedLedgersInBackground`.
> This pip acts inside the method.
>

For the long-running async admin operations such as
deleteTopic/Namespace/Tenant/*, as I mentioned, I think we better provide
the describe* APIs to enable the admins to check the status of the async
operation.

I believe we can first use this system topic from this pip to support case
4, "admin async operations."
Still, we probably need a separate pip to discuss
- the expected behavior/experience of async admin operations.
- if we want to provide describe* APIs for the async operations.
- target resources
- architecture/components how to support describe* APIs

Re: Pulsar Flaky test report 2023-02-07 for PR builds in CI

[Lari Hotari]

Thanks for compiling the report Nicolo.

What are the 10 most flaky tests? It wasn't obvious to me from the email.
I'd like to suggest that we prioritize fixing the 10 most flaky ones.

-Lari

ti 7. helmik. 2023 klo 17.43 Nicolò Boschi  kirjoitti:

> Dear community,
> Here's a report of the flaky tests in Pulsar CI during the observation
> period of 2023-01-31 - 2023-02-07
>
>
> https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-01-31-to-2023-02-07
>
> More flaky test issues:
>
> https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
>
>
> To coordinate the work,
> 1) please search for an existing issues or search for all flaky issues with
> "flaky" or the test class name (without package) in the search:
>
> https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> 2) If there isn't an issue for a particular flaky test failure that you'd
> like to fix, please create an issue using the "Flaky test" template at
> https://github.com/apache/pulsar/issues/new/choose
> 3) Please comment on the issue that you are working on.
>
> I'm looking forward to more contributors joining the efforts. Please join
> the #testing channel on Slack if you'd like to ask questions and tips about
> reproducing flaky tests locally and how to fix them.
> Sharing stories about fixing flaky tests is also helpful for sharing the
> knowledge about how flaky tests can be fixed. That's also a valuable way to
> contribute.
> Some flaky tests might be actual real production code bugs. Fixing
> the flaky test might result in fixing a real production code bug.
>
> More contributions are welcome! Please keep up the good work!
>
> P.S.
> The flaky test reporter was originally coded by Lari. Lately, we simplified
> the way to generate the report to allow more people to access the report
> and for scheduling purpose.
> There are instructions on how to generate it here:
> https://github.com/nicoloboschi/pulsar-flakes#generate-a-new-report
>
>
> Thanks,
> Nicolò Boschi
>

Re: [DISCUSS] Redundant ServiceUrlProvider design and improper use of PIP-121

[Baodi Shi]

 Hi, Asaf.

Maybe?
>
> client.update(ConnectionInfo.builder()
>  .setServiceUrl()
>  .set...()
>  .build())

Yes, This is clearer.


Thanks,
Baodi Shi


在 2023年2月7日 23:48:03 上，Asaf Mesika  写道：

> Oh, thanks Baodi - now I understand what he meant by "calling rebuild".
> I agree, it's not very intuitive.
>
> Maybe?
>
> client.update(ConnectionInfo.builder()
>  .setServiceUrl()
>  .set...()
>  .build())
>
>
> On Mon, Feb 6, 2023 at 2:39 AM Baodi Shi  wrote:
>
>  I thought about it for a moment, and this is not a good idea.
>
>
> Using newBuild may be understood by the user as recreating the client
>
> object. It is still intuitive to add the `updateXXX` method.
>
>
>
> Thanks,
>
> Baodi Shi
>
>
>
> 在 2023年2月5日 23:01:57 上，Asaf Mesika  写道：
>
>
> > Can you please write rebuild example?
>
> >
>
> > On Sun, Feb 5, 2023 at 4:05 PM Baodi Shi  wrote:
>
> >
>
> > >
>
> >
>
> > > Not completely different IMO. When we want to update the service URL,
>
> >
>
> > >
>
> >
>
> > > we usually want to connect to another cluster. However, the cluster
>
> >
>
> > >
>
> >
>
> > > could have configured authentication, TLS for security, but we cannot
>
> >
>
> > >
>
> >
>
> > > guarantee two clusters share the same TLS certificates or
>
> >
>
> > >
>
> >
>
> > > authentication configs. I think we should group these necessary
>
> >
>
> > >
>
> >
>
> > > information into a common class like `ConnectInfo`. Then, just expose
>
> >
>
> > >
>
> >
>
> > > a method to update the `ConnectInfo` so that code could be more
>
> >
>
> > >
>
> >
>
> > > scalable.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > It makes sense to me.
>
> >
>
> >
>
> > Alternatively, can we directly support rebuilding PulsarClient?
>
> >
>
> >
>
> > PulsarClient client = PulsarClient.builder()
>
> >
>
> > .serviceUrl("old url")
>
> >
>
> > .tlsTrustCertsFilePath()
>
> >
>
> > .statsInterval(1, TimeUnit.SECONDS)
>
> >
>
> > .build();
>
> >
>
> >
>
> > client.newBuilder()
>
> >
>
> > .serviceUrl("new url")
>
> >
>
> > // tls relate
>
> >
>
> > .tlsTrustCertsFilePath()
>
> >
>
> > .statsInterval(2, TimeUnit.SECONDS)
>
> >
>
> > .build();
>
> >
>
> >
>
> >
>
> > When calling rebuild, the new configuration overwrites the old
>
> >
>
> > configuration, and if there are no settings, the old configuration is
>
> used.
>
> >
>
> >
>
> > 在 2023年2月2日 23:02:21 上，Asaf Mesika  写道：
>
> >
>
> >
>
> > > Ok, makes sense.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Thu, Feb 2, 2023 at 4:02 PM Yunze Xu 
>
> >
>
> > > wrote:
>
> >
>
> > >
>
> >
>
> > > Not completely different IMO. When we want to update the service URL,
>
> >
>
> > >
>
> >
>
> > > we usually want to connect to another cluster. However, the cluster
>
> >
>
> > >
>
> >
>
> > > could have configured authentication, TLS for security, but we cannot
>
> >
>
> > >
>
> >
>
> > > guarantee two clusters share the same TLS certificates or
>
> >
>
> > >
>
> >
>
> > > authentication configs. I think we should group these necessary
>
> >
>
> > >
>
> >
>
> > > information into a common class like `ConnectInfo`. Then, just expose
>
> >
>
> > >
>
> >
>
> > > a method to update the `ConnectInfo` so that code could be more
>
> >
>
> > >
>
> >
>
> > > scalable.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Thanks,
>
> >
>
> > >
>
> >
>
> > > Yunze
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Thu, Feb 2, 2023 at 9:39 PM Asaf Mesika 
>
> >
>
> > wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Ok, the last two methods IMO are a completely different topic.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > I wonder what the maintainers think about this.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > On Mon, Jan 30, 2023 at 4:16 AM Yunze Xu
>
> 
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > > wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > > In summary, the core issue is that ServiceUrlProvider could be
>
> >
>
> > >
>
> >
>
> > > > > misused. A user might want to implement its own ServiceUrlProvider
>
> by
>
> >
>
> > >
>
> >
>
> > > > > changing the returned value of `get` method but it actually does
>
> not
>
> >
>
> > >
>
> >
>
> > > > > work. The built-in implementations like AutoFailover actually work
>
> by
>
> >
>
> > >
>
> >
>
> > > > > these `updateXXX` methods. So it's meaningless to provide them as
>
> >
>
> > >
>
> >
>
> > > > > ServiceUrlProvider implementations.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > To answer your question directly: totally yes. But not only
>
> >
>
> > >
>
> >
>
> > > > > `setServiceUrl` (the name is `updateServiceUrl` in code),  the
>
> >
>
> > >
>
> >
>
> > > > > `updateAuthentication` and `updateTlsTrustCertsFilePath` methods,
>
> >
>
> > >

Re: [DISCUSS] PIP-245: Subscriptions expiration for NonPersistentTopic only

[Zike Yang]

I agree to remove all subscriptions when there're no consumers on the
non-persistent topic. The configuration
`subscriptionExpirationTimeMinutes ` should not take effect on the
non-persistent topic.

But it seems that currently all Durable subscriptions on
non-persistent topics are NonDurable, and I wonder if that confuses
users.

 > But for `NonPersistentDispatcherMultipleConsumers`,
`NonPersistentStickyKeyDispatcherMultipleConsumers`, I’m not sure that
if it matters when  we auto delete the subscriptions which has no
consumers connected.

It doesn't matter IMO.

Thanks,
Zike Yang



Zike Yang

On Tue, Feb 7, 2023 at 9:22 PM Jiuming Tao
 wrote:
>
> > I think we are talking about the durable subscription of the non-persistent
> > topic, right?
>
> Yes, non-durable subscriptions created by Reader, if the reader disconnected, 
> the subscription will be deleted automatically.
>
>
> For `NonPersistentDispatcher`, it has the following implementations:
> 1. NonPersistentDispatcherMultipleConsumers
> 2. NonPersistentDispatcherSingleActiveConsumer
> 3. NonPersistentStickyKeyDispatcherMultipleConsumers
>
> For `NonPersistentDispatcherSingleActiveConsumer`, auto delete the 
> subscription when there isn’t consumer connected affect nothing.
>
> But for `NonPersistentDispatcherMultipleConsumers`, 
> `NonPersistentStickyKeyDispatcherMultipleConsumers`, I’m not sure that if it 
> matters when  we auto delete the subscriptions which has no consumers 
> connected.
>
> If it doesn’t matter, I agree with you.
>
> Auto delete the durable subscriptions when there aren’t consumers connected 
> to the non-persistent topic.
>
>
> Thanks,
> Tao Jiuming
>
>
> > 2023年2月7日 19:58，PengHui Li mailto:peng...@apache.org>> 
> > 写道：
> >
> > I think we are talking about the durable subscription of the non-persistent
> > topic, right? For the Reader API (non-durable subscription), the
> > subscription will be
> > removed after the reader disconnects. If not, it should be BUG.
> >
> > IMO, we should remove the subscription automatically if all the consumers
> > are disconnected for the non-persistent topic. The consumers can only get
> > the new incoming messages after they connect to the non-persistent topic.
> > It looks like even if we leave the subscription there; it will
> > help nothing.
> > The behavior should be the same as creating a new subscription with the
> > same subscription name.
> >
> > For subscription management, users are not able to disable the subscription
> > auto-creation of the non-persistent topic? We don't have any metadata
> > persist
> > in the metadata store. After the broker restarts, we will lose everything.
> >
> > So I think we don't need to introduce a new configuration. Instead, we
> > should
> > figure out if there is a strong reason to leave the durable subscription
> > after
> > all the consumers are disconnected. If there is no strong reason, I tend to
> > fix
> > incorrect behavior directly.
> >
> > Best,
> > Penghui
> >
> >
> >
> >
> > On Tue, Feb 7, 2023 at 4:14 PM Jiuming Tao  > >
> > wrote:
> >
> >> Hi,
> >>
> >> I’ve opened a new PIP to discuss:
> >>
> >> Currently, in Pulsar, we have a configuration named
> >> `subscriptionExpirationTimeMinutes` to manage the subscription expiration
> >> of `PersistentTopic` and `NonPersistentTopic`.
> >>
> >> When we set a value which is greater than 0 to
> >> `subscriptionExpirationTimeMinutes`, it will affect both `PersistentTopic`
> >> and `NonPersistentTopic`. Their inactive subscriptions will get expired and
> >> will clean automatically.
> >>
> >> For `NonPersistentTopic`, its subscriptions can be clean because we don't
> >> guarantee its data integrity. But for `PersistentTopic`, if we clean its
> >> subscriptions automatically may lead to data loss.
> >>
> >> However, their subscription expiration is managed by the same
> >> configuration(`subscriptionExpirationTimeMinutes`), we can't manage their
> >> subscription expiration independently.
> >>
> >> So I want to introduce a new configuration named
> >> `nonPersistentSubscriptionExpirationTimeMinutes` to manage
> >> `NonPersistentTopic`'s subscription expiration.
> >>
> >>
> >> Please feel free to leave your comments, the PIP link:
> >> https://github.com/apache/pulsar/issues/19448 
> >>  <
> >> https://github.com/apache/pulsar/issues/19448 
> >> >
> >>
> >>
> >> Thanks,
> >> Tao Jiuming
>

Pulsar Cpp client release 3.1.2

[Yunze Xu]

Hi all,

There is a serious OAuth2 authentication regression [1] for all
existing C++ client 3.x.y releases. I see many users tend to downgrade
to old versions like 2.10.2. This behavior is dangerous because of the
CVE.

The fix [2] is now merged, so I decided to start a 3.1.2 release ASAP,
though the 3.1.1 release is not formally announced yet. The Python
3.1.0 release in progress and the Node.js 1.8.1 release will depend on
this version.

[1] https://lists.apache.org/thread/6rrq4lj965rm0zqk9rtwwf6gcqb02220
[2] https://github.com/apache/pulsar-client-cpp/pull/190

Thanks,
Yunze

Re: Pulsar Cpp client release 3.1.2

[Matteo Merli]

+1
--
Matteo Merli


On Tue, Feb 7, 2023 at 6:49 PM Yunze Xu  wrote:
>
> Hi all,
>
> There is a serious OAuth2 authentication regression [1] for all
> existing C++ client 3.x.y releases. I see many users tend to downgrade
> to old versions like 2.10.2. This behavior is dangerous because of the
> CVE.
>
> The fix [2] is now merged, so I decided to start a 3.1.2 release ASAP,
> though the 3.1.1 release is not formally announced yet. The Python
> 3.1.0 release in progress and the Node.js 1.8.1 release will depend on
> this version.
>
> [1] https://lists.apache.org/thread/6rrq4lj965rm0zqk9rtwwf6gcqb02220
> [2] https://github.com/apache/pulsar-client-cpp/pull/190
>
> Thanks,
> Yunze

Re: Pulsar Cpp client release 3.1.2

[Zike Yang]

+1

Thanks,
Zike Yang

On Wed, Feb 8, 2023 at 11:08 AM Matteo Merli  wrote:
>
> +1
> --
> Matteo Merli
> 
>
> On Tue, Feb 7, 2023 at 6:49 PM Yunze Xu  wrote:
> >
> > Hi all,
> >
> > There is a serious OAuth2 authentication regression [1] for all
> > existing C++ client 3.x.y releases. I see many users tend to downgrade
> > to old versions like 2.10.2. This behavior is dangerous because of the
> > CVE.
> >
> > The fix [2] is now merged, so I decided to start a 3.1.2 release ASAP,
> > though the 3.1.1 release is not formally announced yet. The Python
> > 3.1.0 release in progress and the Node.js 1.8.1 release will depend on
> > this version.
> >
> > [1] https://lists.apache.org/thread/6rrq4lj965rm0zqk9rtwwf6gcqb02220
> > [2] https://github.com/apache/pulsar-client-cpp/pull/190
> >
> > Thanks,
> > Yunze

Re: Pulsar Cpp client release 3.1.2

[Michael Marshall]

+1, thank you for fixing this issue, Yunze.

- Michael

On Tue, Feb 7, 2023 at 9:11 PM Zike Yang  wrote:
>
> +1
>
> Thanks,
> Zike Yang
>
> On Wed, Feb 8, 2023 at 11:08 AM Matteo Merli  wrote:
> >
> > +1
> > --
> > Matteo Merli
> > 
> >
> > On Tue, Feb 7, 2023 at 6:49 PM Yunze Xu  
> > wrote:
> > >
> > > Hi all,
> > >
> > > There is a serious OAuth2 authentication regression [1] for all
> > > existing C++ client 3.x.y releases. I see many users tend to downgrade
> > > to old versions like 2.10.2. This behavior is dangerous because of the
> > > CVE.
> > >
> > > The fix [2] is now merged, so I decided to start a 3.1.2 release ASAP,
> > > though the 3.1.1 release is not formally announced yet. The Python
> > > 3.1.0 release in progress and the Node.js 1.8.1 release will depend on
> > > this version.
> > >
> > > [1] https://lists.apache.org/thread/6rrq4lj965rm0zqk9rtwwf6gcqb02220
> > > [2] https://github.com/apache/pulsar-client-cpp/pull/190
> > >
> > > Thanks,
> > > Yunze

Re: [DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Michael Marshall]

This proposal makes sense to me. I am +1 on option B.

Thanks,
Michael

On Wed, Feb 1, 2023 at 2:42 AM Alexander Preuss
 wrote:
>
> Thanks to everyone for providing their input. Please vote on the VOTE
> thread https://lists.apache.org/thread/ykfnfb1pv4bcxwr6lwb4klc3byq71t13
>
> Many thanks,
> Alexander
>
> On Thu, Jan 19, 2023 at 5:01 AM r...@apache.org 
> wrote:
>
> > Hello Alexander, thanks your work for this.
> > It seems that all agrees with this PIP, maybe we can enter the VOTE thread
> > to initiate a VOTE process for this PIP.
> >
> > --
> > Thanks
> > Xiaolong Ran
> >
> > r...@apache.org  于2023年1月19日周四 11:57写道：
> >
> > > +1
> > >
> > > We can imitate the PulsarClient object, and in the BaseContext interface,
> > > add PulsarAdmin and PulsarAdminBuilder as interfaces at the same time.
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> > >
> > > Neng Lu  于2023年1月19日周四 02:28写道：
> > >
> > >> The proposal makes sense to me.
> > >>
> > >> The only note I want to add is to make sure the PulsarAdmin access in
> > >> Source and Sink is also controlled by the flag
> > >> `exposePulsarAdminClientEnabled`. Similar to the following:
> > >>
> > >>
> > >>
> > https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/thread/ThreadRuntimeFactory.java#L111
> > >>
> > >> On 2023/01/03 14:31:52 Alexander Preuss wrote:
> > >> > Hi all,
> > >> >
> > >> > I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO
> > >> > connectors through SinkContext and SourceContext:
> > >> > https://github.com/apache/pulsar/issues/19123
> > >> >
> > >> > Looking forward to hearing your thoughts,
> > >> > Alex
> > >> >
> > >>
> > >
> >

Re: [VOTE] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Michael Marshall]

+1 (binding)

Thanks,
Michael

On Tue, Feb 7, 2023 at 7:00 AM Rui Fu  wrote:
>
> +1
>
> Best,
>
> Rui Fu
> On Jan 30, 2023 at 19:12 +0800, Alexander Preuss 
> , wrote:
> > Now that the DISCUSSION thread has received some more input and some time
> > has passed I would like to start the VOTE again.
> >
> > Best,
> > Alexander
> >
> > On Tue, Jan 17, 2023 at 3:21 AM Bonan Hou 
> > 
> > wrote:
> >
> > > +1
> > >
> > > Bonan Hou
> > >
> > > On Mon, Jan 16, 2023 at 9:56 PM Alexander Preuss
> > >  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm starting the VOTE for PIP-237: Make PulsarAdmin accessible in
> > > > SinkContext and SourceContext:
> > > > https://github.com/apache/pulsar/issues/19123
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/b8r6b4p87qzfgfwmqfql7mqx8kmgorxg
> > > >
> > > > I'd like to go ahead with implementing option B presented in the PIP
> > > > description on GitHub.
> > > >
> > > > Thanks,
> > > > Alexander
> > > >
> > >