Re: [DISCUSS] Dropping the StreamingDispatcher

[Devin Bost]

It looks like the purpose of the StreamingDispatcher was to improve
performance by implementing an improved readahead mechanism. Was it
actually able to accomplish this objective? If so, why do we need a
separate code path for it instead of updating the existing dispatcher to
use the improvement? It's not clear to me if the readahead and
microbatching are mutually exclusive concepts, and microbatching tends to
improve throughput in production, so it seems in theory like both
optimizations would be desirable if they can be combined.

However, I haven't looked deeply into the feasibility of this, so I'd like
to hear more from those with deeper understanding of these code paths.


On Tue, Apr 4, 2023, 2:57 AM Yunze Xu  wrote:

> If the flaky tests were the only concern, I think we can just disable
> these tests. Whatever, this config in `ServiceConfiguration` has
> existed for a long time, though when it was introduced, the PIP rule
> was not clear so there is no PIP for it.
>
> Thanks,
> Yunze
>
> On Tue, Apr 4, 2023 at 3:09 PM Gavin gao  wrote:
> >
> > +1, I totally agree with this idea.
> >
> > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> >
> > > Hello,
> > > It has been a long time that we have in the Pulsar code a new
> > > experimental Dispatcher implementation named StreamingDispatcher.
> > >
> > > https://github.com/apache/pulsar/pull/9056
> > >
> > > There are many flaky tests about that feature and I believe that it
> > > has never been used in Production by anyone, because it happened a few
> > > times that we did some changes in the regular Dispatcher and
> > > introduced bugs on the StreamingDispacther (usually manifested as
> > > flaky tests)
> > >
> > >
> > > I propose to drop the StreamingDispatcher code for Pulsar 3.0.
> > > I don't think we need a PIP for this, it is an experimental code that
> > > was never delivered as a production ready feature.
> > >
> > > If anyone is aware of users please chime in.
> > >
> > > If anyone wants to sponsor that feature and objects in removing this
> > > dead code (that we still have to maintain) please help us in
> > > completing the feature.
> > >
> > > On paper it is a very appealing feature, and I am disappointed in
> dropping
> > > it.
> > > On the other hand, this is dead code that we have to maintain with zero
> > > benefit
> > >
> > > Thoughts ?
> > >
> > > Enrico
> > >
>

Re: [DISCUSS] Dropping the StreamingDispatcher

[Enrico Olivelli]

Yunze,

Il Mar 4 Apr 2023, 09:57 Yunze Xu  ha scritto:

> If the flaky tests were the only concern, I think we can just disable
> these tests.


My concern is not about the the flaky tests but a out maintenance of dead
code.



Whatever, this config in `ServiceConfiguration` has
> existed for a long time, though when it was introduced, the PIP rule
> was not clear so there is no PIP for it.
>

I don't think it would work well in production, given the amount of
flakyness in the tests and the fact that nobody ever asked questions about
it.


This is why I propose to drop the code now in Pulsar 3.0


Enrico



> Thanks,
> Yunze
>
> On Tue, Apr 4, 2023 at 3:09 PM Gavin gao  wrote:
> >
> > +1, I totally agree with this idea.
> >
> > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> >
> > > Hello,
> > > It has been a long time that we have in the Pulsar code a new
> > > experimental Dispatcher implementation named StreamingDispatcher.
> > >
> > > https://github.com/apache/pulsar/pull/9056
> > >
> > > There are many flaky tests about that feature and I believe that it
> > > has never been used in Production by anyone, because it happened a few
> > > times that we did some changes in the regular Dispatcher and
> > > introduced bugs on the StreamingDispacther (usually manifested as
> > > flaky tests)
> > >
> > >
> > > I propose to drop the StreamingDispatcher code for Pulsar 3.0.
> > > I don't think we need a PIP for this, it is an experimental code that
> > > was never delivered as a production ready feature.
> > >
> > > If anyone is aware of users please chime in.
> > >
> > > If anyone wants to sponsor that feature and objects in removing this
> > > dead code (that we still have to maintain) please help us in
> > > completing the feature.
> > >
> > > On paper it is a very appealing feature, and I am disappointed in
> dropping
> > > it.
> > > On the other hand, this is dead code that we have to maintain with zero
> > > benefit
> > >
> > > Thoughts ?
> > >
> > > Enrico
> > >
>

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

[Lari Hotari]

I think we might want to move ExtensibleLoadManagerImplTest to the flaky group 
until the fixes have been completed. Most builds fail now because of the 
flakiness of that test. That is blocking our progress and moving to flaky group 
would resolve that issue.
Once that it is proven to be stable, we can make a simple update to change the 
test group.
I'll proceed with a PR for this soon.

-Lari

On 2023/04/04 16:39:21 Heesung Sohn wrote:
> Hi,
> 
> Thank you for the report.
> 
> We are fixing the errors from ExtensibleLoadManagerImplTest.
> 
> Regards,
> Heesung
> 
> On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi  wrote:
> 
> > Dear community,
> > Here's a report of the flaky tests in Pulsar CI during the observation
> > period of 2023-03-27 - 2023-04-03
> >
> >
> > https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-03-27-to-2023-04-03
> >
> >
> > Most flaky tests:
> >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.stateCheck
> > 9
> >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testSplitBundleAdminAPI
> > 4
> >
> > org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
> > 4
> >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState
> > 3
> >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign
> > 2
> > More flaky test issues:
> >
> > https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
> >
> >
> > To coordinate the work,
> > 1) please search for an existing issues or search for all flaky issues with
> > "flaky" or the test class name (without package) in the search:
> >
> > https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> > 2) If there isn't an issue for a particular flaky test failure that you'd
> > like to fix, please create an issue using the "Flaky test" template at
> > https://github.com/apache/pulsar/issues/new/choose
> > 3) Please comment on the issue that you are working on.
> >
> >
> > More contributions are welcome!
> > Nicolò Boschi
> >
> 

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

[Heesung Sohn]

Hi,

The fix PR( #19945 ) has been
merged(I also notified here,
https://github.com/apache/pulsar/issues/20007#issuecomment-1497679571).
If this test continues to be flaky, I agree with quarantining it not to
harm builds for the time being.

Regards,
Heesung



On Wed, Apr 5, 2023 at 6:24 AM Lari Hotari  wrote:

> I think we might want to move ExtensibleLoadManagerImplTest to the flaky
> group until the fixes have been completed. Most builds fail now because of
> the flakiness of that test. That is blocking our progress and moving to
> flaky group would resolve that issue.
> Once that it is proven to be stable, we can make a simple update to change
> the test group.
> I'll proceed with a PR for this soon.
>
> -Lari
>
> On 2023/04/04 16:39:21 Heesung Sohn wrote:
> > Hi,
> >
> > Thank you for the report.
> >
> > We are fixing the errors from ExtensibleLoadManagerImplTest.
> >
> > Regards,
> > Heesung
> >
> > On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi 
> wrote:
> >
> > > Dear community,
> > > Here's a report of the flaky tests in Pulsar CI during the observation
> > > period of 2023-03-27 - 2023-04-03
> > >
> > >
> > >
> https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-03-27-to-2023-04-03
> > >
> > >
> > > Most flaky tests:
> > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.stateCheck
> > > 9
> > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testSplitBundleAdminAPI
> > > 4
> > >
> > >
> org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
> > > 4
> > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState
> > > 3
> > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign
> > > 2
> > > More flaky test issues:
> > >
> > >
> https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
> > >
> > >
> > > To coordinate the work,
> > > 1) please search for an existing issues or search for all flaky issues
> with
> > > "flaky" or the test class name (without package) in the search:
> > >
> > >
> https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> > > 2) If there isn't an issue for a particular flaky test failure that
> you'd
> > > like to fix, please create an issue using the "Flaky test" template at
> > > https://github.com/apache/pulsar/issues/new/choose
> > > 3) Please comment on the issue that you are working on.
> > >
> > >
> > > More contributions are welcome!
> > > Nicolò Boschi
> > >
> >
>

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

[Lari Hotari]

Thank you Heesung for resolving the problem so quickly.

-Lari

ke 5. huhtik. 2023 klo 20.23 Heesung Sohn
 kirjoitti:

> Hi,
>
> The fix PR( #19945 ) has been
> merged(I also notified here,
> https://github.com/apache/pulsar/issues/20007#issuecomment-1497679571).
> If this test continues to be flaky, I agree with quarantining it not to
> harm builds for the time being.
>
> Regards,
> Heesung
>
>
>
> On Wed, Apr 5, 2023 at 6:24 AM Lari Hotari  wrote:
>
> > I think we might want to move ExtensibleLoadManagerImplTest to the flaky
> > group until the fixes have been completed. Most builds fail now because
> of
> > the flakiness of that test. That is blocking our progress and moving to
> > flaky group would resolve that issue.
> > Once that it is proven to be stable, we can make a simple update to
> change
> > the test group.
> > I'll proceed with a PR for this soon.
> >
> > -Lari
> >
> > On 2023/04/04 16:39:21 Heesung Sohn wrote:
> > > Hi,
> > >
> > > Thank you for the report.
> > >
> > > We are fixing the errors from ExtensibleLoadManagerImplTest.
> > >
> > > Regards,
> > > Heesung
> > >
> > > On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi 
> > wrote:
> > >
> > > > Dear community,
> > > > Here's a report of the flaky tests in Pulsar CI during the
> observation
> > > > period of 2023-03-27 - 2023-04-03
> > > >
> > > >
> > > >
> >
> https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-03-27-to-2023-04-03
> > > >
> > > >
> > > > Most flaky tests:
> > > >
> > > >
> >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.stateCheck
> > > > 9
> > > >
> > > >
> >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testSplitBundleAdminAPI
> > > > 4
> > > >
> > > >
> >
> org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
> > > > 4
> > > >
> > > >
> >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState
> > > > 3
> > > >
> > > >
> >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign
> > > > 2
> > > > More flaky test issues:
> > > >
> > > >
> >
> https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
> > > >
> > > >
> > > > To coordinate the work,
> > > > 1) please search for an existing issues or search for all flaky
> issues
> > with
> > > > "flaky" or the test class name (without package) in the search:
> > > >
> > > >
> >
> https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> > > > 2) If there isn't an issue for a particular flaky test failure that
> > you'd
> > > > like to fix, please create an issue using the "Flaky test" template
> at
> > > > https://github.com/apache/pulsar/issues/new/choose
> > > > 3) Please comment on the issue that you are working on.
> > > >
> > > >
> > > > More contributions are welcome!
> > > > Nicolò Boschi
> > > >
> > >
> >
>

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

[Michael Marshall]

Thanks for fixing that test Heesung, and thanks for the report Nicolo!

- Michael

On Wed, Apr 5, 2023 at 2:17 PM Lari Hotari  wrote:
>
> Thank you Heesung for resolving the problem so quickly.
>
> -Lari
>
> ke 5. huhtik. 2023 klo 20.23 Heesung Sohn
>  kirjoitti:
>
> > Hi,
> >
> > The fix PR( #19945 ) has been
> > merged(I also notified here,
> > https://github.com/apache/pulsar/issues/20007#issuecomment-1497679571).
> > If this test continues to be flaky, I agree with quarantining it not to
> > harm builds for the time being.
> >
> > Regards,
> > Heesung
> >
> >
> >
> > On Wed, Apr 5, 2023 at 6:24 AM Lari Hotari  wrote:
> >
> > > I think we might want to move ExtensibleLoadManagerImplTest to the flaky
> > > group until the fixes have been completed. Most builds fail now because
> > of
> > > the flakiness of that test. That is blocking our progress and moving to
> > > flaky group would resolve that issue.
> > > Once that it is proven to be stable, we can make a simple update to
> > change
> > > the test group.
> > > I'll proceed with a PR for this soon.
> > >
> > > -Lari
> > >
> > > On 2023/04/04 16:39:21 Heesung Sohn wrote:
> > > > Hi,
> > > >
> > > > Thank you for the report.
> > > >
> > > > We are fixing the errors from ExtensibleLoadManagerImplTest.
> > > >
> > > > Regards,
> > > > Heesung
> > > >
> > > > On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi 
> > > wrote:
> > > >
> > > > > Dear community,
> > > > > Here's a report of the flaky tests in Pulsar CI during the
> > observation
> > > > > period of 2023-03-27 - 2023-04-03
> > > > >
> > > > >
> > > > >
> > >
> > https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-03-27-to-2023-04-03
> > > > >
> > > > >
> > > > > Most flaky tests:
> > > > >
> > > > >
> > >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.stateCheck
> > > > > 9
> > > > >
> > > > >
> > >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testSplitBundleAdminAPI
> > > > > 4
> > > > >
> > > > >
> > >
> > org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
> > > > > 4
> > > > >
> > > > >
> > >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState
> > > > > 3
> > > > >
> > > > >
> > >
> > org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign
> > > > > 2
> > > > > More flaky test issues:
> > > > >
> > > > >
> > >
> > https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
> > > > >
> > > > >
> > > > > To coordinate the work,
> > > > > 1) please search for an existing issues or search for all flaky
> > issues
> > > with
> > > > > "flaky" or the test class name (without package) in the search:
> > > > >
> > > > >
> > >
> > https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> > > > > 2) If there isn't an issue for a particular flaky test failure that
> > > you'd
> > > > > like to fix, please create an issue using the "Flaky test" template
> > at
> > > > > https://github.com/apache/pulsar/issues/new/choose
> > > > > 3) Please comment on the issue that you are working on.
> > > > >
> > > > >
> > > > > More contributions are welcome!
> > > > > Nicolò Boschi
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] Dropping the StreamingDispatcher

[Michael Marshall]

If the code isn't being used or maintained, I support removing it. The
code will be available in the git history in case someone decides to
resurrect it.

Thanks,
Michael

On Wed, Apr 5, 2023 at 7:14 AM Enrico Olivelli  wrote:
>
> Yunze,
>
> Il Mar 4 Apr 2023, 09:57 Yunze Xu  ha scritto:
>
> > If the flaky tests were the only concern, I think we can just disable
> > these tests.
>
>
> My concern is not about the the flaky tests but a out maintenance of dead
> code.
>
>
>
> Whatever, this config in `ServiceConfiguration` has
> > existed for a long time, though when it was introduced, the PIP rule
> > was not clear so there is no PIP for it.
> >
>
> I don't think it would work well in production, given the amount of
> flakyness in the tests and the fact that nobody ever asked questions about
> it.
>
>
> This is why I propose to drop the code now in Pulsar 3.0
>
>
> Enrico
>
>
>
> > Thanks,
> > Yunze
> >
> > On Tue, Apr 4, 2023 at 3:09 PM Gavin gao  wrote:
> > >
> > > +1, I totally agree with this idea.
> > >
> > > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> > >
> > > > Hello,
> > > > It has been a long time that we have in the Pulsar code a new
> > > > experimental Dispatcher implementation named StreamingDispatcher.
> > > >
> > > > https://github.com/apache/pulsar/pull/9056
> > > >
> > > > There are many flaky tests about that feature and I believe that it
> > > > has never been used in Production by anyone, because it happened a few
> > > > times that we did some changes in the regular Dispatcher and
> > > > introduced bugs on the StreamingDispacther (usually manifested as
> > > > flaky tests)
> > > >
> > > >
> > > > I propose to drop the StreamingDispatcher code for Pulsar 3.0.
> > > > I don't think we need a PIP for this, it is an experimental code that
> > > > was never delivered as a production ready feature.
> > > >
> > > > If anyone is aware of users please chime in.
> > > >
> > > > If anyone wants to sponsor that feature and objects in removing this
> > > > dead code (that we still have to maintain) please help us in
> > > > completing the feature.
> > > >
> > > > On paper it is a very appealing feature, and I am disappointed in
> > dropping
> > > > it.
> > > > On the other hand, this is dead code that we have to maintain with zero
> > > > benefit
> > > >
> > > > Thoughts ?
> > > >
> > > > Enrico
> > > >
> >

Re: [DISCUSS] Dropping the StreamingDispatcher

[Devin Bost]

+1 since it can be pulled back up in git history if someone decides to do
something with it to improve it at a later time.

I also agree that it's a pain to maintain, and I don't know anyone using
it. I've gone through some of those code paths, and I was concerned about
divergence anyway.

- Devin


On Wed, Apr 5, 2023, 5:40 PM Michael Marshall  wrote:

> If the code isn't being used or maintained, I support removing it. The
> code will be available in the git history in case someone decides to
> resurrect it.
>
> Thanks,
> Michael
>
> On Wed, Apr 5, 2023 at 7:14 AM Enrico Olivelli 
> wrote:
> >
> > Yunze,
> >
> > Il Mar 4 Apr 2023, 09:57 Yunze Xu  ha
> scritto:
> >
> > > If the flaky tests were the only concern, I think we can just disable
> > > these tests.
> >
> >
> > My concern is not about the the flaky tests but a out maintenance of dead
> > code.
> >
> >
> >
> > Whatever, this config in `ServiceConfiguration` has
> > > existed for a long time, though when it was introduced, the PIP rule
> > > was not clear so there is no PIP for it.
> > >
> >
> > I don't think it would work well in production, given the amount of
> > flakyness in the tests and the fact that nobody ever asked questions
> about
> > it.
> >
> >
> > This is why I propose to drop the code now in Pulsar 3.0
> >
> >
> > Enrico
> >
> >
> >
> > > Thanks,
> > > Yunze
> > >
> > > On Tue, Apr 4, 2023 at 3:09 PM Gavin gao 
> wrote:
> > > >
> > > > +1, I totally agree with this idea.
> > > >
> > > > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> > > >
> > > > > Hello,
> > > > > It has been a long time that we have in the Pulsar code a new
> > > > > experimental Dispatcher implementation named StreamingDispatcher.
> > > > >
> > > > > https://github.com/apache/pulsar/pull/9056
> > > > >
> > > > > There are many flaky tests about that feature and I believe that it
> > > > > has never been used in Production by anyone, because it happened a
> few
> > > > > times that we did some changes in the regular Dispatcher and
> > > > > introduced bugs on the StreamingDispacther (usually manifested as
> > > > > flaky tests)
> > > > >
> > > > >
> > > > > I propose to drop the StreamingDispatcher code for Pulsar 3.0.
> > > > > I don't think we need a PIP for this, it is an experimental code
> that
> > > > > was never delivered as a production ready feature.
> > > > >
> > > > > If anyone is aware of users please chime in.
> > > > >
> > > > > If anyone wants to sponsor that feature and objects in removing
> this
> > > > > dead code (that we still have to maintain) please help us in
> > > > > completing the feature.
> > > > >
> > > > > On paper it is a very appealing feature, and I am disappointed in
> > > dropping
> > > > > it.
> > > > > On the other hand, this is dead code that we have to maintain with
> zero
> > > > > benefit
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Enrico
> > > > >
> > >
>

Re: Pulsar Flaky test report 2023-04-03 for PR builds in CI

[Devin Bost]

Great team work on all the flaky tests. It's much appreciated.

- Devin

On Wed, Apr 5, 2023, 4:39 PM Michael Marshall  wrote:

> Thanks for fixing that test Heesung, and thanks for the report Nicolo!
>
> - Michael
>
> On Wed, Apr 5, 2023 at 2:17 PM Lari Hotari  wrote:
> >
> > Thank you Heesung for resolving the problem so quickly.
> >
> > -Lari
> >
> > ke 5. huhtik. 2023 klo 20.23 Heesung Sohn
> >  kirjoitti:
> >
> > > Hi,
> > >
> > > The fix PR( #19945 ) has
> been
> > > merged(I also notified here,
> > > https://github.com/apache/pulsar/issues/20007#issuecomment-1497679571
> ).
> > > If this test continues to be flaky, I agree with quarantining it not to
> > > harm builds for the time being.
> > >
> > > Regards,
> > > Heesung
> > >
> > >
> > >
> > > On Wed, Apr 5, 2023 at 6:24 AM Lari Hotari  wrote:
> > >
> > > > I think we might want to move ExtensibleLoadManagerImplTest to the
> flaky
> > > > group until the fixes have been completed. Most builds fail now
> because
> > > of
> > > > the flakiness of that test. That is blocking our progress and moving
> to
> > > > flaky group would resolve that issue.
> > > > Once that it is proven to be stable, we can make a simple update to
> > > change
> > > > the test group.
> > > > I'll proceed with a PR for this soon.
> > > >
> > > > -Lari
> > > >
> > > > On 2023/04/04 16:39:21 Heesung Sohn wrote:
> > > > > Hi,
> > > > >
> > > > > Thank you for the report.
> > > > >
> > > > > We are fixing the errors from ExtensibleLoadManagerImplTest.
> > > > >
> > > > > Regards,
> > > > > Heesung
> > > > >
> > > > > On Tue, Apr 4, 2023 at 3:01 AM Nicolò Boschi  >
> > > > wrote:
> > > > >
> > > > > > Dear community,
> > > > > > Here's a report of the flaky tests in Pulsar CI during the
> > > observation
> > > > > > period of 2023-03-27 - 2023-04-03
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> https://github.com/nicoloboschi/pulsar-flakes/tree/master/2023-03-27-to-2023-04-03
> > > > > >
> > > > > >
> > > > > > Most flaky tests:
> > > > > >
> > > > > >
> > > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.stateCheck
> > > > > > 9
> > > > > >
> > > > > >
> > > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testSplitBundleAdminAPI
> > > > > > 4
> > > > > >
> > > > > >
> > > >
> > >
> org.apache.bookkeeper.mledger.impl.ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors
> > > > > > 4
> > > > > >
> > > > > >
> > > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.initializeState
> > > > > > 3
> > > > > >
> > > > > >
> > > >
> > >
> org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign
> > > > > > 2
> > > > > > More flaky test issues:
> > > > > >
> > > > > >
> > > >
> > >
> https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue
> > > > > >
> > > > > >
> > > > > > To coordinate the work,
> > > > > > 1) please search for an existing issues or search for all flaky
> > > issues
> > > > with
> > > > > > "flaky" or the test class name (without package) in the search:
> > > > > >
> > > > > >
> > > >
> > >
> https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> > > > > > 2) If there isn't an issue for a particular flaky test failure
> that
> > > > you'd
> > > > > > like to fix, please create an issue using the "Flaky test"
> template
> > > at
> > > > > > https://github.com/apache/pulsar/issues/new/choose
> > > > > > 3) Please comment on the issue that you are working on.
> > > > > >
> > > > > >
> > > > > > More contributions are welcome!
> > > > > > Nicolò Boschi
> > > > > >
> > > > >
> > > >
> > >
>

[DISCUSS] Release Pulsar Node.js Client 1.8.2

[Baodi Shi]

Hi everyone,

I would like to propose releasing the Pulsar Node.js Client v1.8.2.

The current latest version (v1.8.1) has a serious problem


   - It does not run in Node.js version greater than 16 (Linux
   environment)[0].
   - Also, it's not compatible with Alpine 3.15[1].



[0] https://github.com/apache/pulsar-client-node/pull/310
[1] https://github.com/apache/pulsar-client-node/issues/299



Thanks,
Baodi Shi

Re: [DISCUSS] Dropping the StreamingDispatcher

[mattisonchao]

Totally agree with it. +1

Best
Mattison
On Apr 6, 2023, 10:53 +0800, Devin Bost , wrote:
> +1 since it can be pulled back up in git history if someone decides to do
> something with it to improve it at a later time.
>
> I also agree that it's a pain to maintain, and I don't know anyone using
> it. I've gone through some of those code paths, and I was concerned about
> divergence anyway.
>
> - Devin
>
>
> On Wed, Apr 5, 2023, 5:40 PM Michael Marshall  wrote:
>
> > If the code isn't being used or maintained, I support removing it. The
> > code will be available in the git history in case someone decides to
> > resurrect it.
> >
> > Thanks,
> > Michael
> >
> > On Wed, Apr 5, 2023 at 7:14 AM Enrico Olivelli 
> > wrote:
> > > >
> > > > Yunze,
> > > >
> > > > Il Mar 4 Apr 2023, 09:57 Yunze Xu  ha
> > scritto:
> > > >
> > > > > > If the flaky tests were the only concern, I think we can just 
> > > > > > disable
> > > > > > these tests.
> > > >
> > > >
> > > > My concern is not about the the flaky tests but a out maintenance of 
> > > > dead
> > > > code.
> > > >
> > > >
> > > >
> > > > Whatever, this config in `ServiceConfiguration` has
> > > > > > existed for a long time, though when it was introduced, the PIP rule
> > > > > > was not clear so there is no PIP for it.
> > > > > >
> > > >
> > > > I don't think it would work well in production, given the amount of
> > > > flakyness in the tests and the fact that nobody ever asked questions
> > about
> > > > it.
> > > >
> > > >
> > > > This is why I propose to drop the code now in Pulsar 3.0
> > > >
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > > >
> > > > > > On Tue, Apr 4, 2023 at 3:09 PM Gavin gao 
> > wrote:
> > > > > > > >
> > > > > > > > +1, I totally agree with this idea.
> > > > > > > >
> > > > > > > > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > > It has been a long time that we have in the Pulsar code a 
> > > > > > > > > > new
> > > > > > > > > > experimental Dispatcher implementation named 
> > > > > > > > > > StreamingDispatcher.
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/pulsar/pull/9056
> > > > > > > > > >
> > > > > > > > > > There are many flaky tests about that feature and I believe 
> > > > > > > > > > that it
> > > > > > > > > > has never been used in Production by anyone, because it 
> > > > > > > > > > happened a
> > few
> > > > > > > > > > times that we did some changes in the regular Dispatcher and
> > > > > > > > > > introduced bugs on the StreamingDispacther (usually 
> > > > > > > > > > manifested as
> > > > > > > > > > flaky tests)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I propose to drop the StreamingDispatcher code for Pulsar 
> > > > > > > > > > 3.0.
> > > > > > > > > > I don't think we need a PIP for this, it is an experimental 
> > > > > > > > > > code
> > that
> > > > > > > > > > was never delivered as a production ready feature.
> > > > > > > > > >
> > > > > > > > > > If anyone is aware of users please chime in.
> > > > > > > > > >
> > > > > > > > > > If anyone wants to sponsor that feature and objects in 
> > > > > > > > > > removing
> > this
> > > > > > > > > > dead code (that we still have to maintain) please help us in
> > > > > > > > > > completing the feature.
> > > > > > > > > >
> > > > > > > > > > On paper it is a very appealing feature, and I am 
> > > > > > > > > > disappointed in
> > > > > > dropping
> > > > > > > > > > it.
> > > > > > > > > > On the other hand, this is dead code that we have to 
> > > > > > > > > > maintain with
> > zero
> > > > > > > > > > benefit
> > > > > > > > > >
> > > > > > > > > > Thoughts ?
> > > > > > > > > >
> > > > > > > > > > Enrico
> > > > > > > > > >
> > > > > >
> >

Re: [VOTE] PIP-257: Add Open ID Connect Support to Server Components

[Michael Marshall]

+ 1 (binding)

PIP passes with 4 binding +1 votes (Lari, Dave, Enrico, and Michael)
and 1 non-binding +1 (Devin)

There is some final feedback I need to address on the PIP, and then
I'll merge the two PRs.

Thanks,
Michael

On Tue, Mar 28, 2023 at 11:00 AM Devin Bost  wrote:
>
> +1
>
> Devin G. Bost
>
>
> On Tue, Mar 28, 2023 at 1:57 AM Enrico Olivelli  wrote:
>
> > +1 (binding)
> >
> > Enrico
> >
> > Il giorno mar 28 mar 2023 alle ore 04:59 Dave Fisher
> >  ha scritto:
> > >
> > > +1 (binding)
> > >
> > > Sent from my iPhone
> > >
> > > > On Mar 27, 2023, at 12:59 PM, Lari Hotari  wrote:
> > > >
> > > > +1, binding
> > > >
> > > > -Lari
> > > >
> > > >> On Mon, 27 Mar 2023, 22.37 Michael Marshall, 
> > wrote:
> > > >>
> > > >> Hi Pulsar Community,
> > > >>
> > > >> This thread is to start the vote for PIP 257.
> > > >>
> > > >> Discussion:
> > > >> https://lists.apache.org/thread/ttdh74t6v5nznmtw436otk8qdymjvrfk
> > > >> Issue: https://github.com/apache/pulsar/issues/19771
> > > >> Work in Progress Implementations:
> > > >> - https://github.com/apache/pulsar/pull/19849
> > > >> - https://github.com/apache/pulsar/pull/19888
> > > >>
> > > >> The implementation still has some minor details to iron out, but I
> > > >> don't think those are a large enough issue to delay the vote any
> > > >> longer. If you feel otherwise, please let me know and continue the
> > > >> discussion on the discussion thread.
> > > >>
> > > >> Thanks,
> > > >> Michael
> > > >>
> > >
> >

Re: [VOTE] PIP-250: Add proxyVersion to CommandConnect

[Michael Marshall]

+1 (binding)

PIP passes with 3 binding +1 votes (Enrico, Yunze, Michael) and 4 non
binding +1 votes (ZhangJian, Yubiao, Zike, Zixuan)

Thanks,
Michael

On Mon, Mar 13, 2023 at 5:22 AM Zixuan Liu  wrote:
>
> +1 (non-binding)
>
> Thanks,
> Zixuan
>
> Yunze Xu  于2023年3月13日周一 17:31写道：
>
> > +1 (binding)
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Mar 13, 2023 at 3:48 PM Zike Yang  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Thanks
> > > Zike Yang
> > >
> > > On Sat, Mar 11, 2023 at 12:08 PM Yubiao Feng
> > >  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > > > On Sat, Mar 11, 2023 at 3:31 AM Michael Marshall  > >
> > > > wrote:
> > > >
> > > > > Hello Pulsar Community,
> > > > >
> > > > > This thread is to start the vote for PIP 250.
> > > > >
> > > > > Discussion:
> > > > > https://lists.apache.org/thread/ntzwkdz23d6s6ycyn2r198c515npp7vt
> > > > > Issue: https://github.com/apache/pulsar/issues/19623
> > > > > Work in Progress Implementation:
> > > > > https://github.com/apache/pulsar/pull/19618
> > > > >
> > > > > I'll update the implementation to align with Enrico's proposal on the
> > > > > discussion thread.
> > > > >
> > > > > Voting typically stays open for at least 48h. Since I am starting
> > this
> > > > > on a Friday, I plan to keep it open for at least 96 hours (ending on
> > > > > Tuesday).
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> >