[GitHub] [pulsar-adapters] aditiwari01 commented on issue #29: Pulsar - Spark adapter for scala 2.11

2021-12-20 Thread GitBox


aditiwari01 commented on issue #29:
URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-997713940


   The current available spark pulsar adapter seems to be unreliable. I've 
written a reliable connector with addition consideration for back-pressure and 
rate limit. Currently testing around it. Can raise a PR if that's fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




Re: Status of Pulsar 2.9.0 and starting 2.9.1

2021-12-20 Thread Yu
Hi Enrico,

Thanks for your contribution.

1) A soft reminder that is not on the current release process [a]:
Since 2.9.0 was delayed, some doc changes on the master are applied to
2.10.0 only, so generating the 2.9.0 doc set should be based on the "2.9.0
release time point" rather than the current master, or else the 2.9.0 doc
set might contain 2.10 feature docs.

2) 2.9.0 is not announced and the 2.9.0 release note is available on the
website.
For the previous releases, we have tech blogs for them to (announce and)
explain highlights [b], will you create a tech blog for 2.9.0?

[a] https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
[b] https://pulsar.apache.org/blog/2021/09/23/Apache-Pulsar-2-8-1/


On Fri, Dec 17, 2021 at 7:36 PM Enrico Olivelli  wrote:

> Il giorno ven 17 dic 2021 alle ore 09:45 Yu  ha scritto:
>
> > Hi Enrico,
> >
> > Thanks for your great effort on the 2.9.0 release.
> >
> > Circling back to see if there is any progress of the 2.9.0 website
> updates
> > [1].
> >
> > Currently, the 2.9.0 doc is not available on the website and the 2.9.0
> doc
> > set has not been generated yet, any updates? Thanks
> >
>
>
> I will do it together with 2.9.1
> because 2.9.0 is broken, so it is better to not do much buzz around it
>
> Enrico
>
>
> >
> > [1]
> > https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
> >
> >
> > On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli 
> > wrote:
> >
> > > Dave,
> > > You are correct.
> > > Pulsar 2.9.0 has already been released and also some people already
> > started
> > > to report issues.
> > > The docker images have been deployed and we cannot change them.
> > >
> > > I am finishing the release process for 2.9.0 with the website updates.
> > >
> > > I am preparing 2.9.1.
> > >
> > > I propose to just skip the announcement for 2.9.0.
> > >
> > > If we are quick during the VOTE we can close this story within the end
> of
> > > the week
> > >
> > > Enrico
> > >
> > > Il Lun 13 Dic 2021, 06:34 Dave Fisher  ha
> > scritto:
> > >
> > > > (1) we have published 2.9.0 at
> > > > https://downloads.apache.org/pulsar/pulsar-2.9.0/
> > > >
> > > > (2) we have published 2.9.0 artifacts through maven central. They
> don’t
> > > > let anyone republish versions.
> > > >
> > > > There are no do overs on versions. We simply cannot redo 2.9.0 at
> this
> > > > moment.
> > > >
> > > > All the best,
> > > > Dave
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo  wrote:
> > > > >
> > > > > My take is - if we haven't announced 2.9, I would suggest just
> > redoing
> > > > the
> > > > > 2.9.0 release.
> > > > >
> > > > > - Sijie
> > > > >
> > > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen 
> > > wrote:
> > > > >>
> > > > >> I am a little confused about why we should skip 2.9.0 and not
> > continue
> > > > >> to release 2.9.0 by including the critical bug fixes. In fact, the
> > > > >> Pulsar 2.9.0 release is not yet completed.
> > > > >>
> > > > >> For users, they will worry about whether the Pulsar release
> process
> > is
> > > > >> standardized if we skip 2.9.0. They will also worry about the
> > release
> > > > >> quality of Apache Pulsar if we have found the critical bugs before
> > it
> > > > >> is released but not included it into the release version. For
> Pulsar
> > > > >> 2.9.0, it couldn't be deployed into the production environment due
> > to
> > > > >> the critical bug https://github.com/apache/pulsar/pull/12993
> > > > >>
> > > > >> Regards,
> > > > >> Hang
> > > > >>
> > > > >> Dave Fisher  于2021年12月13日周一 09:40写道:
> > > > >>>
> > > > >>> It can be the case that releases are not announced. For example
> > with
> > > > >> Tomcat a version which fails to pass the vote is skipped.
> > > > >>>
> > > > >>> Let’s not announce 2.9.0 and go on to 2.9.1.
> > > > >>>
> > > > >>> Maybe there’s some website fixes to hide 2.9.0.
> > > > >>>
> > > > >>>
> > > > >>> Sent from my iPhone
> > > > >>>
> > > >  On Dec 12, 2021, at 5:28 PM, PengHui Li 
> > wrote:
> > > > 
> > > >  Another point is we have not announced the 2.9.0 release yet.
> > > > 
> > > >  This will make users feel confused that a new release from the
> > > Pulsar
> > > >  community with the
> > > >  serious problem(log4j bug) but after the log4j has fixed the
> issue
> > > and
> > > >  provided the new release.
> > > > 
> > > >  I think we'd better contain the fix in 2.9.0 and 2.9.0 also has
> a
> > > > >> critical
> > > >  bug https://github.com/apache/pulsar/pull/12993
> > > >  which will lead the topic stop to provide services for more than
> > > 5min.
> > > > >> It
> > > >  looks like, hey, we have a new release here but
> > > >  it has critical security issues and known serious bugs which
> will
> > > > >> seriously
> > > >  affect the core features.
> > > > 
> > > >  From the perspective of release, yes, the release vote has
> closed.
> > > But
> > > > >> I
> > 

Re: Status of Pulsar 2.9.0 and starting 2.9.1

2021-12-20 Thread Enrico Olivelli
Yu,

Il giorno lun 20 dic 2021 alle ore 10:01 Yu  ha scritto:

> Hi Enrico,
>
> Thanks for your contribution.
>
> 1) A soft reminder that is not on the current release process [a]:
> Since 2.9.0 was delayed, some doc changes on the master are applied to
> 2.10.0 only, so generating the 2.9.0 doc set should be based on the "2.9.0
> release time point" rather than the current master, or else the 2.9.0 doc
> set might contain 2.10 feature docs.
>

This is actually a good point.
I am not sure about how I can fix this.
Maybe I can  try to create the versioned docs from the v2.9.0 tag



>
> 2) 2.9.0 is not announced and the 2.9.0 release note is available on the
> website.
> For the previous releases, we have tech blogs for them to (announce and)
> explain highlights [b], will you create a tech blog for 2.9.0?
>

I am sorry but I won't have time to do this before the beginning of January,
it would be useful if someone could do it

Enrico



>
> [a]
> https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
> [b] https://pulsar.apache.org/blog/2021/09/23/Apache-Pulsar-2-8-1/
>
>
> On Fri, Dec 17, 2021 at 7:36 PM Enrico Olivelli 
> wrote:
>
> > Il giorno ven 17 dic 2021 alle ore 09:45 Yu  ha
> scritto:
> >
> > > Hi Enrico,
> > >
> > > Thanks for your great effort on the 2.9.0 release.
> > >
> > > Circling back to see if there is any progress of the 2.9.0 website
> > updates
> > > [1].
> > >
> > > Currently, the 2.9.0 doc is not available on the website and the 2.9.0
> > doc
> > > set has not been generated yet, any updates? Thanks
> > >
> >
> >
> > I will do it together with 2.9.1
> > because 2.9.0 is broken, so it is better to not do much buzz around it
> >
> > Enrico
> >
> >
> > >
> > > [1]
> > >
> https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
> > >
> > >
> > > On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli 
> > > wrote:
> > >
> > > > Dave,
> > > > You are correct.
> > > > Pulsar 2.9.0 has already been released and also some people already
> > > started
> > > > to report issues.
> > > > The docker images have been deployed and we cannot change them.
> > > >
> > > > I am finishing the release process for 2.9.0 with the website
> updates.
> > > >
> > > > I am preparing 2.9.1.
> > > >
> > > > I propose to just skip the announcement for 2.9.0.
> > > >
> > > > If we are quick during the VOTE we can close this story within the
> end
> > of
> > > > the week
> > > >
> > > > Enrico
> > > >
> > > > Il Lun 13 Dic 2021, 06:34 Dave Fisher  ha
> > > scritto:
> > > >
> > > > > (1) we have published 2.9.0 at
> > > > > https://downloads.apache.org/pulsar/pulsar-2.9.0/
> > > > >
> > > > > (2) we have published 2.9.0 artifacts through maven central. They
> > don’t
> > > > > let anyone republish versions.
> > > > >
> > > > > There are no do overs on versions. We simply cannot redo 2.9.0 at
> > this
> > > > > moment.
> > > > >
> > > > > All the best,
> > > > > Dave
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo 
> wrote:
> > > > > >
> > > > > > My take is - if we haven't announced 2.9, I would suggest just
> > > redoing
> > > > > the
> > > > > > 2.9.0 release.
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen 
> > > > wrote:
> > > > > >>
> > > > > >> I am a little confused about why we should skip 2.9.0 and not
> > > continue
> > > > > >> to release 2.9.0 by including the critical bug fixes. In fact,
> the
> > > > > >> Pulsar 2.9.0 release is not yet completed.
> > > > > >>
> > > > > >> For users, they will worry about whether the Pulsar release
> > process
> > > is
> > > > > >> standardized if we skip 2.9.0. They will also worry about the
> > > release
> > > > > >> quality of Apache Pulsar if we have found the critical bugs
> before
> > > it
> > > > > >> is released but not included it into the release version. For
> > Pulsar
> > > > > >> 2.9.0, it couldn't be deployed into the production environment
> due
> > > to
> > > > > >> the critical bug https://github.com/apache/pulsar/pull/12993
> > > > > >>
> > > > > >> Regards,
> > > > > >> Hang
> > > > > >>
> > > > > >> Dave Fisher  于2021年12月13日周一 09:40写道:
> > > > > >>>
> > > > > >>> It can be the case that releases are not announced. For example
> > > with
> > > > > >> Tomcat a version which fails to pass the vote is skipped.
> > > > > >>>
> > > > > >>> Let’s not announce 2.9.0 and go on to 2.9.1.
> > > > > >>>
> > > > > >>> Maybe there’s some website fixes to hide 2.9.0.
> > > > > >>>
> > > > > >>>
> > > > > >>> Sent from my iPhone
> > > > > >>>
> > > > >  On Dec 12, 2021, at 5:28 PM, PengHui Li 
> > > wrote:
> > > > > 
> > > > >  Another point is we have not announced the 2.9.0 release yet.
> > > > > 
> > > > >  This will make users feel confused that a new release from the
> > > > Pulsar
> > > > >  community with the
> > > > >  serious problem(log4j bug) but after the log4j has fixed the
> > issue
> > > > an

[DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

2021-12-20 Thread Zike Yang
https://github.com/apache/pulsar/issues/13408

Pasted below for quoting convenience.

——


## Motivation

If we enable the DLQ when consuming messages. For some messages that can't be 
processed successfully, the messages will be moved to the DLQ, but if we do not 
specify the data retention for the namespace or create a subscription for the 
DLQ to retain the data, the data of the DLQ topic will be removed 
automatically. Therefore, we need to create the initial subscription before 
sending messages to the DLQ.

## Goal

Users can set the initial subscription name in the DeadLetterPolicy. The 
consumer will create the initial subscription before sending messages to the 
DLQ. At this point, subsequent messages produced to the DLQ are not 
automatically deleted unexpectedly.

This PIP needs to be compatible with the previous behavior. The initial 
subscription name in the DeadLetterPolicy is optional. Default value is the 
subscription name of the consumer.

## API Changes

Add `initSubscriptionName` to the `DeadLetterPolicy`

```java
/**
 * Name of the initial subscription name of the dead letter topic.
 * The default value is the subscription name of the consumer.
 */
private String initSubscriptionName;
```

## Implementation

Before the deadLetterProducer is initialized, the consumer first tries to 
create a deadLetterConsumer using the initial subscription name in the 
DeadLetterPolicy. When this subscription already exists, the ConsumerBusy 
exception will occur. In this case, we can ignore that exception and create the 
deadLetterProducer.

Prototype implementation PR: https://github.com/apache/pulsar/pull/13355



Thanks,
Zike Yang



Re: [VOTE] Pulsar Release 2.7.4 Candidate 1

2021-12-20 Thread Nozomi Kurihara
+1 (binding)

- verified checksums and signatures
- build from source
- verified produce/consume and functions
- verified cassandra connector
- verified stateful function

Thanks,
Nozomi

2021年12月18日(土) 19:43 Enrico Olivelli :

> Jiwei Guo,
> Thanks for driving the release
> today is Saturday, so I am not sure how many people will have time to test
> the release candidate during the 72 hours period (for instance I can do it
> only on Monday, hopefully).
> Please take this into consideration when you are going to close the VOTE,
> maybe waiting until Tuesday may be a good idea.
>
> Enrico
>
>
>
> Il giorno sab 18 dic 2021 alle ore 09:54 guo jiwei 
> ha scritto:
>
> > This is the first release candidate for Apache Pulsar, version 2.7.4.
> >
> > It fixes the following issues:
> >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
> >
> > Release note:
> > https://github.com/apache/pulsar/pull/13391
> >
> > *** Please download, test and vote on this release. This vote will
> > stay openfor at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-1/
> >
> > SHA-512 checksums:
> >
> >
> >
> 1a34a408085328500c8c4c72e5ca9b458058a088674b1bfd550fe01ad365182b8820516f99b5c010a03037e1886ce5539cb57f5ef4a2297f1b7ed8fb5844bcd0
> >  apache-pulsar-2.7.4-bin.tar.gz
> >
> >
> 37e75698946f3390a254fa2bd7d974fca27d1aed632e86d64a08dc0bf8f2e0ca830cbeed2820782e59ac6bac7368ae69b54f1f5cd3d6c2cad292166e3538e285
> >  apache-pulsar-2.7.4-src.tar.gz
> >
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1113
> >
> > The tag to be voted upon:
> > v2.7.4-candidate-1 (ab451b855d873a9bad2005f939a23118a583baa9)
> > https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
>


Re: [apache/pulsar] [website] website upgrade (#11766)

2021-12-20 Thread Yu
Hi Dave,

Thanks for your instructions.

+1 for not migrating the incubating versioned docs since they are too old
and almost no one uses them. Besides, as you suggested, we can regenerate
those docs if needed.

On Mon, Dec 20, 2021 at 11:18 AM Dave Fisher  wrote:

> This is a question for the dev@pulsar list and not a blocker for
> infrastructure.
>
> I don’t think there is an overriding requirement to preserve docs from
> incubating releases. If there were then there are other ways to find and
> regenerate these docs.
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Dec 19, 2021, at 6:58 PM, Anonymitaet 
> wrote:
> >
> > 
> > Hi @dave2wave Thanks for your answer. However,
> https://archive.apache.org/dist/incubator/pulsar/ hosts code files rather
> than doc files. We are looking for doc files (source files) of these
> versions, do you know where hosts them? Thanks again
> >
> > 2.0.1-incubating
> > 2.0.0-rc1-incubating
> > 1.22.0-incubating
> > 1.21.0-incubating
> > 1.20.0-incubating
> > 1.19.0-incubating
> > —
> > Reply to this email directly, view it on GitHub, or unsubscribe.
> > Triage notifications on the go with GitHub Mobile for iOS or Android.
> > You are receiving this because you were mentioned.
>


[VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-20 Thread linlin
This is the second release candidate for Apache Pulsar, version 2.8.2

It fixes the following issues:
https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+label%3Arelease%2F2.8.2+is%3Aclosed

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.
Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-2/

SHA-512 checksums:
59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
 apache-pulsar-2.8.2-bin.tar.gz

82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
 apache-pulsar-2.8.2-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1129/

The tag to be voted upon:
v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Release notes draft:
https://github.com/apache/pulsar/pull/13400

Please download the source package, and follow the README to build
and run the Pulsar standalone service.


Re: [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

2021-12-20 Thread PengHui Li
Hi Enrico,

Have you checked the CI status after completing the 2.9.1 PRs
cherry-picking?

Looks this test failed when I run the test with tag 2.9.1.

```
[INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
[ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time elapsed:
26.782 s <<< FAILURE! - in
org.apache.pulsar.broker.transaction.TransactionTest
[ERROR]
testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
 Time elapsed: 0.107 s  <<< FAILURE!
java.lang.AssertionError: expected [4] but found [3]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.testng.Assert.assertEquals(Assert.java:917)
at
org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at
org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at
org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at
org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
[INFO]   Run 1: PASS
[ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
expected [4] but found [3]
[INFO]
[INFO]
[ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
```

Regards,
Penghui

On Mon, Dec 20, 2021 at 3:06 PM Enrico Olivelli  wrote:

> Hello everyone,
> The VOTE passed with 5 VOTEs, 3 of them were binding.
>
> - Matteo Merli (binding)
> - Enrico Olivelli (binding)
> - Peng Hui (binding)
> - Nicolò Boschi
> - Massimiliano Mirelli
>
> I will move forward with the next steps
> https://github.com/apache/pulsar/wiki/Release-process
>
> Enrico
>
>
> Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li 
> ha
> scritto:
>
> > +1 binding
> >
> > Penghui
> >
> > Enrico Olivelli 于2021年12月18日 周六18:39写道:
> >
> > > +1 (binding)
> > >
> > > - Run release validation procedure
> > > - CI is passing on those sources
> > >
> > > Enrico
> > >
> > > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li <
> peng...@apache.org>
> > > ha
> > > scritto:
> > >
> > > > >  Will this issue be fixed in the future releases?
> > > >
> > > > Yes, 2.8.2 and 2.9.2 will fix the problem.
> > > >
> > > > Penghui
> > > >
> > > > On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> > > > massimilianomirelli...@gmail.com> wrote:
> > > >
> > > > > Thank you for the rc!
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * verify sha512 checksums
> > > > > * verify gpg signatures
> > > > > * build pulsar-all docker image
> > > > > * execute Fallout distributed system test (produce / receive 10k
> > > > messages)
> > > > >
> > > > > Building the docker image as indicated in the README:
> > > > >
> > > > > mvn clean install -DskipTests
> > > > > mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
> > > > >
> > > > > I still get the error described in this PR#11951 (
> > > > > https://github.com/apache/pulsar/pull/11951) which I suppose has
> > been
> > > > > cherry picked in 2.9.1.
> > > > >
> > > > > Giving enough permissions to /docker/pulsar/scripts/ in the src
> > package
> > > > and
> > > > > then building the docker image again solved the issue.
> > > > >
> > > > > I also tested the problem with the image Enrico provided, this way:
> > > > >
> > > > > docker run -it --entrypoint bash eolivelli/pulsar-all:2.9.1rc2
> > > > > ls -al /pulsar/bin
> > > > >
> > > > > and that one does have the right permissions.
> > > > >
> > > > > Enrico, did you build the image using the same mvn commands (^^^),
> or
> > > is
> > > > > there some other way to build it?
> > > > >
> > > > > Thank you,
> > > > > Max
> > > > >
> > > > > On Fri, 17 Dec 2021 at 16:18, 陳智弘  wrote:
> > > > >
> > > > > > Hi PengHu,
> > > > > >
> > > > > >  Will this issue be fixed in the future releases?
> > > > > >
> > > > > > PengHui Li  於 2021年12月17日 週五 21:53 寫道:
> > > > > >
> > > > > > > Hi Enrico,
> > > > > > >
> > > > > > > I'm ok, it only happens when the message is without a schema
> > > version.
> > > > > > >

Re: [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

2021-12-20 Thread Enrico Olivelli
Il giorno lun 20 dic 2021 alle ore 13:21 PengHui Li  ha
scritto:

> Hi Enrico,
>
> Have you checked the CI status after completing the 2.9.1 PRs
> cherry-picking?
>

When I created the tag I am pretty sure that CI on GH actions passed.

I hope that no-one committed something to branch-2.9

If this happens, then we must enforce some rules about the fact that only
the RM can commit to a branch under release.

In ZooKeeper we create a "release branch" in order to prevent such problems

Enrico



>
> Looks this test failed when I run the test with tag 2.9.1.
>
> ```
> [INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time elapsed:
> 26.782 s <<< FAILURE! - in
> org.apache.pulsar.broker.transaction.TransactionTest
> [ERROR]
>
> testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>  Time elapsed: 0.107 s  <<< FAILURE!
> java.lang.AssertionError: expected [4] but found [3]
> at org.testng.Assert.fail(Assert.java:99)
> at org.testng.Assert.failNotEquals(Assert.java:1037)
> at org.testng.Assert.assertEqualsImpl(Assert.java:140)
> at org.testng.Assert.assertEquals(Assert.java:122)
> at org.testng.Assert.assertEquals(Assert.java:907)
> at org.testng.Assert.assertEquals(Assert.java:917)
> at
>
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
> at
>
> org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
> at
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
> at
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]
>
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
> [INFO]   Run 1: PASS
> [ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
> expected [4] but found [3]
> [INFO]
> [INFO]
> [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
> ```
>
> Regards,
> Penghui
>
> On Mon, Dec 20, 2021 at 3:06 PM Enrico Olivelli 
> wrote:
>
> > Hello everyone,
> > The VOTE passed with 5 VOTEs, 3 of them were binding.
> >
> > - Matteo Merli (binding)
> > - Enrico Olivelli (binding)
> > - Peng Hui (binding)
> > - Nicolò Boschi
> > - Massimiliano Mirelli
> >
> > I will move forward with the next steps
> > https://github.com/apache/pulsar/wiki/Release-process
> >
> > Enrico
> >
> >
> > Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li 
> > ha
> > scritto:
> >
> > > +1 binding
> > >
> > > Penghui
> > >
> > > Enrico Olivelli 于2021年12月18日 周六18:39写道:
> > >
> > > > +1 (binding)
> > > >
> > > > - Run release validation procedure
> > > > - CI is passing on those sources
> > > >
> > > > Enrico
> > > >
> > > > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li <
> > peng...@apache.org>
> > > > ha
> > > > scritto:
> > > >
> > > > > >  Will this issue be fixed in the future releases?
> > > > >
> > > > > Yes, 2.8.2 and 2.9.2 will fix the problem.
> > > > >
> > > > > Penghui
> > > > >
> > > > > On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> > > > > massimilianomirelli...@gmail.com> wrote:
> > > > >
> > > > > > Thank you for the rc!
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > * verify sha512 checksums
> > > > > > * verify gpg signatures
> > > > > > * build pulsar-all docker image
> > > > > > * execute Fallout distributed system test (produce / receive 10k
> > > > > messages)
> > > > > >
> > > > > > Building the docker image as indicated in the README:
> > > > > >
> > > > > > mvn clean install -DskipTests
> > > > > > mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
> > > > > >
> > > > > > I still get the error described in this PR#11951 (
> > > > > > https://github.com/apache/pulsar/pull/11951) which I suppose has
> > > been
> > > > > > cherry picked in 2.9.1.
> > > > > >
> > > > > > Giving enough permissions to /docker/pulsar/scripts/ in the src
> > > package
> > > > > and
> > > > > > then building the docker image again solved the issue.
> > > > > >
> > > > > > I also tested the problem with the image Enrico provided, this
> way:
> > > > > >
> > > > > > docker run -it --entrypoint bash eolivelli

Re: [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

2021-12-20 Thread PengHui Li
I use the 2.9.1 tag to run the test, not branch-2.9.

I think this one(2db23b8dd3eb859cc16e30686578be275026c347) is the last one
that you cherry-picked?

But there are failed tests:

[image: image.png]

Penghui

On Mon, Dec 20, 2021 at 8:24 PM Enrico Olivelli  wrote:

> Il giorno lun 20 dic 2021 alle ore 13:21 PengHui Li 
> ha
> scritto:
>
> > Hi Enrico,
> >
> > Have you checked the CI status after completing the 2.9.1 PRs
> > cherry-picking?
> >
>
> When I created the tag I am pretty sure that CI on GH actions passed.
>
> I hope that no-one committed something to branch-2.9
>
> If this happens, then we must enforce some rules about the fact that only
> the RM can commit to a branch under release.
>
> In ZooKeeper we create a "release branch" in order to prevent such problems
>
> Enrico
>
>
>
> >
> > Looks this test failed when I run the test with tag 2.9.1.
> >
> > ```
> > [INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
> > [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time elapsed:
> > 26.782 s <<< FAILURE! - in
> > org.apache.pulsar.broker.transaction.TransactionTest
> > [ERROR]
> >
> >
> testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
> >  Time elapsed: 0.107 s  <<< FAILURE!
> > java.lang.AssertionError: expected [4] but found [3]
> > at org.testng.Assert.fail(Assert.java:99)
> > at org.testng.Assert.failNotEquals(Assert.java:1037)
> > at org.testng.Assert.assertEqualsImpl(Assert.java:140)
> > at org.testng.Assert.assertEquals(Assert.java:122)
> > at org.testng.Assert.assertEquals(Assert.java:907)
> > at org.testng.Assert.assertEquals(Assert.java:917)
> > at
> >
> >
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> >
> >
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
> > at
> >
> >
> org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
> > at
> >
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
> > at
> >
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Failures:
> > [ERROR]
> >
> >
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
> > [INFO]   Run 1: PASS
> > [ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
> > expected [4] but found [3]
> > [INFO]
> > [INFO]
> > [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
> > ```
> >
> > Regards,
> > Penghui
> >
> > On Mon, Dec 20, 2021 at 3:06 PM Enrico Olivelli 
> > wrote:
> >
> > > Hello everyone,
> > > The VOTE passed with 5 VOTEs, 3 of them were binding.
> > >
> > > - Matteo Merli (binding)
> > > - Enrico Olivelli (binding)
> > > - Peng Hui (binding)
> > > - Nicolò Boschi
> > > - Massimiliano Mirelli
> > >
> > > I will move forward with the next steps
> > > https://github.com/apache/pulsar/wiki/Release-process
> > >
> > > Enrico
> > >
> > >
> > > Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li <
> peng...@apache.org>
> > > ha
> > > scritto:
> > >
> > > > +1 binding
> > > >
> > > > Penghui
> > > >
> > > > Enrico Olivelli 于2021年12月18日 周六18:39写道:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > - Run release validation procedure
> > > > > - CI is passing on those sources
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li <
> > > peng...@apache.org>
> > > > > ha
> > > > > scritto:
> > > > >
> > > > > > >  Will this issue be fixed in the future releases?
> > > > > >
> > > > > > Yes, 2.8.2 and 2.9.2 will fix the problem.
> > > > > >
> > > > > > Penghui
> > > > > >
> > > > > > On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> > > > > > massimilianomirelli...@gmail.com> wrote:
> > > > > >
> > > > > > > Thank you for the rc!
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > * verify sha512 checksums
> > > > > > > * verify gpg signatures
> > > > > > > * build pulsar-all docker image
> > > > > > > * execute Fallout distributed system test (produce / receive
> 10k
> > > > > > messages)
> > > > > > >
> > > > > > > Building the docker image as indicated in the README:
> > > > > > >
> > > > > > > mvn clean install -DskipTests
> > > > 

DISCUSS - [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

2021-12-20 Thread Enrico Olivelli
(changing the subject)


Il giorno lun 20 dic 2021 alle ore 13:33 PengHui Li  ha
scritto:

> I use the 2.9.1 tag to run the test, not branch-2.9.
>
> I think this one(2db23b8dd3eb859cc16e30686578be275026c347) is the last one
> that you cherry-picked?
>

This is the latest commit: f52ac045f41acbb6c31da21a3463df3cfbe8f1b4

And you are correct, this is the latest commit I cherry-picked:
2db23b8dd3eb859cc16e30686578be275026c347 [Managed Ledger] Fix the incorrect
total size when BrokerEntryMetadata is enabled (#12714)

for reference:
 https://github.com/apache/pulsar/tree/v2.9.1

commit f52ac045f41acbb6c31da21a3463df3cfbe8f1b4 (HEAD -> branch-2.9, tag:
v2.9.1-candidate-2, tag: v2.9.1, origin/branch-2.9)
Author: Nicolò Boschi 
Date:   Wed Dec 15 22:18:58 2021 +0100
[security] Upgrade Netty to 4.1.72 - CVE-2021-43797 (#13328)


Enrico


But there are failed tests:
>
> [image: image.png]
>
> Penghui
>
> On Mon, Dec 20, 2021 at 8:24 PM Enrico Olivelli 
> wrote:
>
>> Il giorno lun 20 dic 2021 alle ore 13:21 PengHui Li 
>> ha
>> scritto:
>>
>> > Hi Enrico,
>> >
>> > Have you checked the CI status after completing the 2.9.1 PRs
>> > cherry-picking?
>> >
>>
>> When I created the tag I am pretty sure that CI on GH actions passed.
>>
>> I hope that no-one committed something to branch-2.9
>>
>> If this happens, then we must enforce some rules about the fact that only
>> the RM can commit to a branch under release.
>>
>> In ZooKeeper we create a "release branch" in order to prevent such
>> problems
>>
>> Enrico
>>
>>
>>
>> >
>> > Looks this test failed when I run the test with tag 2.9.1.
>> >
>> > ```
>> > [INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
>> > [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time
>> elapsed:
>> > 26.782 s <<< FAILURE! - in
>> > org.apache.pulsar.broker.transaction.TransactionTest
>> > [ERROR]
>> >
>> >
>> testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>> >  Time elapsed: 0.107 s  <<< FAILURE!
>> > java.lang.AssertionError: expected [4] but found [3]
>> > at org.testng.Assert.fail(Assert.java:99)
>> > at org.testng.Assert.failNotEquals(Assert.java:1037)
>> > at org.testng.Assert.assertEqualsImpl(Assert.java:140)
>> > at org.testng.Assert.assertEquals(Assert.java:122)
>> > at org.testng.Assert.assertEquals(Assert.java:907)
>> > at org.testng.Assert.assertEquals(Assert.java:917)
>> > at
>> >
>> >
>> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:498)
>> > at
>> >
>> >
>> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
>> > at
>> >
>> >
>> org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
>> > at
>> >
>> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
>> > at
>> >
>> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> > [INFO]
>> > [INFO] Results:
>> > [INFO]
>> > [ERROR] Failures:
>> > [ERROR]
>> >
>> >
>> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>> > [INFO]   Run 1: PASS
>> > [ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
>> > expected [4] but found [3]
>> > [INFO]
>> > [INFO]
>> > [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
>> > ```
>> >
>> > Regards,
>> > Penghui
>> >
>> > On Mon, Dec 20, 2021 at 3:06 PM Enrico Olivelli 
>> > wrote:
>> >
>> > > Hello everyone,
>> > > The VOTE passed with 5 VOTEs, 3 of them were binding.
>> > >
>> > > - Matteo Merli (binding)
>> > > - Enrico Olivelli (binding)
>> > > - Peng Hui (binding)
>> > > - Nicolò Boschi
>> > > - Massimiliano Mirelli
>> > >
>> > > I will move forward with the next steps
>> > > https://github.com/apache/pulsar/wiki/Release-process
>> > >
>> > > Enrico
>> > >
>> > >
>> > > Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li <
>> peng...@apache.org>
>> > > ha
>> > > scritto:
>> > >
>> > > > +1 binding
>> > > >
>> > > > Penghui
>> > > >
>> > > > Enrico Olivelli 于2021年12月18日 周六18:39写道:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > - Run release validation procedure
>> > > > > - CI is passing on those sources
>> > > > >
>> > > > > Enrico
>> > > > >
>> > > > > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li

[GitHub] [pulsar-adapters] eolivelli commented on issue #29: Pulsar - Spark adapter for scala 2.11

2021-12-20 Thread GitBox


eolivelli commented on issue #29:
URL: https://github.com/apache/pulsar-adapters/issues/29#issuecomment-998040594


   > Can raise a PR if that's fine
   
   Sure! thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-20 Thread PengHui Li
+1 (binding)

Check signature,
Run standalone,
Verify Cassandra connector,
Verify Function
Verify PusarSQL, still have `java.nio.BufferUnderflowException`, but it
only happens on the topic created by the Function,
I have tried to publish new string messages to a new topic, and query data
from the topic, it works. It's not a regression in 2.8.2

Penghui

On Mon, Dec 20, 2021 at 7:19 PM linlin  wrote:

> This is the second release candidate for Apache Pulsar, version 2.8.2
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+label%3Arelease%2F2.8.2+is%3Aclosed
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-2/
>
> SHA-512 checksums:
>
> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
>  apache-pulsar-2.8.2-bin.tar.gz
>
>
> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
>  apache-pulsar-2.8.2-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1129/
>
> The tag to be voted upon:
> v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
> https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Release notes draft:
> https://github.com/apache/pulsar/pull/13400
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>


[DISCUSS] Remove Grafana and Prometheus Dockerfiles; Remove DCOS example

2021-12-20 Thread Michael Marshall
Hi Pulsar Community,

I would like to clean up our main project's `docker/` directory. We
have a couple of old Dockerfiles that either need to be updated or
removed. My vote is to remove them.

1. Grafana: with every release, we build and upload a custom grafana
docker image just to package dashboards. Grafana provides other
mechanisms to load dashboards. I propose we remove the grafana image
from our release process and then move our dashboard json files to a
new top level directory named `grafana/`. [0]

Our image is based on grafana/grafana:4.3.1. The current grafana
release is grafana/grafana:8.3.3.

2. Prometheus: we have a prometheus Dockerfile that was initially
merged as part of a DCOS example 4 years ago. We have never published
this docker image as part of a release [1]. There is nothing pulsar
specific in these files. [2]

Our image is based on prom/prometheus:v1.8.2. The latest prometheus
image prom/prometheus:v2.32.1.

3. Can we remove our DCOS examples in `deployment/dcos`? We haven't
updated our DCOS example deployment spec in 3.5 years. The current
spec relies on our custom grafana docker image and on a community
member's custom build [3] of our prometheus-dcos docker image. I don't
think our example deployment should rely on a community member's
custom docker image. (I don't have a PR to remove this example yet.)

Alternatively, we could update the dockerfiles and the DCOS example.
However, the lack of activity in keeping these three project
dependencies up to date indicates to me that we should remove them.

Let me know what you think.

Thanks,
Michael

[0] https://github.com/apache/pulsar/pull/13389
[1] https://hub.docker.com/u/apachepulsar/
[2] https://github.com/apache/pulsar/pull/13387
[3] 
https://github.com/apache/pulsar/blob/c62ca808fc8c5aed3bb96b99bf6ef0c8ed382e3f/deployment/dcos/PulsarGroups.json#L251


[GitHub] [pulsar-client-node] tschmidt64-SC commented on issue #134: Getting error while installing

2021-12-20 Thread GitBox


tschmidt64-SC commented on issue #134:
URL: 
https://github.com/apache/pulsar-client-node/issues/134#issuecomment-998219865


   I'm still having this issue to this day. Tried building in a docker 
container, same issue there. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




Re: [DISCUSS] Release Pulsar 2.7.4

2021-12-20 Thread Michael Marshall
Thank you Penghui, Technoboy, and BewareMyPower for picking up where I
left off to get the 2.7.4 release ready to go. You cherry-picked many commits,
resolved many conflicts, and fixed a bunch of tests!

Thanks,
Michael


On Thu, Dec 16, 2021 at 7:37 PM guo jiwei  wrote:
>
> Hi,
> After we have fixed some issue like ZookeeperCache NPE, listing namespace
> exception, and skip some flaky tests (verified locally), now the CI have
> passed.
> Skipped flaky tests are tracked here:
> https://github.com/apache/pulsar/issues/13299
> Now we decide to vote for releasing 2.7.4.
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Tue, Dec 14, 2021 at 11:58 AM PengHui Li  wrote:
>
> > Thanks for the update, I will move it 2.7.5
> >
> > Thanks,
> > Penghui
> >
> > On Tue, Dec 14, 2021 at 9:47 AM Matteo Merli 
> > wrote:
> >
> > > Let's take https://github.com/apache/pulsar/pull/12484 out of the
> > > picture since it's failing the tests.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Sun, Dec 12, 2021 at 11:06 PM PengHui Li  wrote:
> > > >
> > > > Yes,
> > > >
> > > > https://github.com/apache/pulsar/pull/13215 has cherry-picked, so we
> > can
> > > > close it.
> > > > https://github.com/apache/pulsar/pull/12484 blocked by the test.
> > > >
> > > > Penghui
> > > >
> > > > On Mon, Dec 13, 2021 at 2:35 PM Dave Fisher 
> > > wrote:
> > > >
> > > > > I see 2 PRs still open at
> > > > >
> > >
> > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.7.4
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Dec 12, 2021, at 8:22 PM, guo jiwei 
> > wrote:
> > > > > >
> > > > > > I have pushed out some fixes in
> > > > > https://github.com/apache/pulsar/pull/13243
> > > > > > After the tests get passed, I will send out the RC-1 VOTE for 2.7.4
> > > > > >
> > > > > > Regards
> > > > > > Jiwei Guo (Tboy)
> > > > > >
> > > > > >
> > > > > >> On Sun, Dec 12, 2021 at 3:11 PM PengHui Li 
> > > wrote:
> > > > > >>
> > > > > >> Just put an update here. We have done the PR cherry-picking
> > > > > >>
> > > > > >> https://github.com/apache/pulsar/commits/branch-2.7
> > > > > >>
> > > > > >> And most of the integration tests are fixed due to the docker
> > image
> > > > > issue
> > > > > >> or the testcontainer issue, now some integration tests get passed,
> > > but
> > > > > some
> > > > > >> are not.
> > > > > >> And there are some failed tests, maybe a flaky test, we need to
> > > ensure
> > > > > it's
> > > > > >> not a regression.
> > > > > >>
> > > > > >> We are continuing on the test part.
> > > > > >>
> > > > > >> Penghui
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>> On Sat, Dec 11, 2021 at 5:36 PM PengHui Li 
> > > wrote:
> > > > > >>>
> > > > > >>> Hi Michael,
> > > > > >>>
> > > > > >>> +1,
> > > > > >>>
> > > > > >>> Thanks for the great work.
> > > > > >>> We will continue on the PR cherry-picking and the release process
> > > to
> > > > > make
> > > > > >>> sure the urgent release can be done ASAP.
> > > > > >>>
> > > > > >>> Penghui
> > > > > >>>
> > > > > >>> On Sat, Dec 11, 2021 at 3:42 PM Michael Marshall <
> > > mmarsh...@apache.org
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Given the log4j CVE, we should work to release 2.7.4.
> > > > > 
> > > > >  I started preparing the release today by cherry-picking merged
> > PRs
> > > > >  that have the `release/2.7.4` label but have not yet been
> > > > >  cherry-picked to `branch-2.7` [0]. There are still 37 PRs that
> > > have
> > > > >  not been cherry picked. I think it will take too long to cherry
> > > pick
> > > > >  all of these commits, as many have conflicts, and we should
> > > prioritize
> > > > >  releasing 2.7.4. The main commits that we should get
> > cherry-picked
> > > > >  before creating the git tag are any labeled with
> > > `component/security`.
> > > > >  There are only a few remaining commits to cherry pick. Please
> > let
> > > me
> > > > >  know if you think any other commits ought to be cherry-picked.
> > > > > 
> > > > >  The earliest I'll be able to build the release is Monday. If we
> > > need
> > > > >  to start sooner, perhaps someone else will be available to
> > manage
> > > this
> > > > >  urgent release.
> > > > > 
> > > > >  Thanks,
> > > > >  Michael
> > > > > 
> > > > >  [0] -
> > > > > 
> > > > > >>
> > > > >
> > >
> > https://github.com/apache/pulsar/pulls?page=2&q=label%3Arelease%2F2.7.4+sort%3Acreated-asc+is%3Apr+-label%3Acherry-picked%2Fbranch-2.7
> > > > >  [1] -
> > > > > 
> > > > > >>
> > > > >
> > >
> > https://github.com/apache/pulsar/pulls?q=label%3Arelease%2F2.7.4+sort%3Acreated-asc+is%3Apr+-label%3Acherry-picked%2Fbranch-2.7+label%3Acomponent%2Fsecurity
> > > > > 
> > > > > 
> > > > >  On Thu, Dec 9, 2021 at 4:03 PM Neng Lu 
> > wrote:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > On 2021/12/09 15:29:55 Michael Marshall wrote:
> > > > > >> H

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

2021-12-20 Thread Michael Marshall
Thanks for creating this PIP Zike Yang. This sounds like an important
improvement. I have a couple thoughts.

1. Does the `retryLetterTopic` have the same problem with immediate
message expiration? Based on a quick glance, I don't see anything
creating a subscription for the retry topic.

2. Your prototype exposes a lacking in our existing client
implementation: the non-admin client cannot directly create
subscriptions. IIUC, this implementation does not work if
`allowAutoSubscriptionCreation` is false. Further, it is inefficient
to create and immediately close a consumer just to create a
subscription. Perhaps we should just update the non-admin client so
that it can create subscriptions? It could be private so that we don't
create confusion with the non-admin and the admin clients.

Thanks,
Michael


On Mon, Dec 20, 2021 at 4:18 AM Zike Yang
 wrote:
>
> https://github.com/apache/pulsar/issues/13408
>
> Pasted below for quoting convenience.
>
> ——
>
>
> ## Motivation
>
> If we enable the DLQ when consuming messages. For some messages that can't be 
> processed successfully, the messages will be moved to the DLQ, but if we do 
> not specify the data retention for the namespace or create a subscription for 
> the DLQ to retain the data, the data of the DLQ topic will be removed 
> automatically. Therefore, we need to create the initial subscription before 
> sending messages to the DLQ.
>
> ## Goal
>
> Users can set the initial subscription name in the DeadLetterPolicy. The 
> consumer will create the initial subscription before sending messages to the 
> DLQ. At this point, subsequent messages produced to the DLQ are not 
> automatically deleted unexpectedly.
>
> This PIP needs to be compatible with the previous behavior. The initial 
> subscription name in the DeadLetterPolicy is optional. Default value is the 
> subscription name of the consumer.
>
> ## API Changes
>
> Add `initSubscriptionName` to the `DeadLetterPolicy`
>
> ```java
> /**
>  * Name of the initial subscription name of the dead letter topic.
>  * The default value is the subscription name of the consumer.
>  */
> private String initSubscriptionName;
> ```
>
> ## Implementation
>
> Before the deadLetterProducer is initialized, the consumer first tries to 
> create a deadLetterConsumer using the initial subscription name in the 
> DeadLetterPolicy. When this subscription already exists, the ConsumerBusy 
> exception will occur. In this case, we can ignore that exception and create 
> the deadLetterProducer.
>
> Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
>
>
>
> Thanks,
> Zike Yang
>


Re: [DISCUSSION] PIP-120: Enable client memory limit by default

2021-12-20 Thread Michael Marshall
> I think it's a good time for 2.10 to enable this setting by default and,
> correspondingly, to disable by default the producer queue size limit.

+1

> 64MB is picked because it's a small enough memory size that will guarantee
> a very high producer throughput, irrespective of the individual messages size.

Did you consider determining the default limit by inspecting the JVM's runtime?
For example, in the broker, we set `maxMessagePublishBufferSizeInMB` to 1/2
of direct memory, by default.

I concede that this config is more complicated than a broker config since
we're talking about a client application. However, it seems like applications
with 1 GB memory would probably want a different setting than one with
32 GB memory, and a fractional value could make the client work better
"out of the box."

We cannot make everyone happy with a default value, so if this dynamic
default is too complicated to implement, I'm good with 64 MB. :)

Thanks,
Michael





On Sun, Dec 19, 2021 at 11:17 PM Lan Liang  wrote:
>
> +1
>
>
> On 12/17/2021 14:43,mattison chao wrote:
> +1
>
> On Fri, 17 Dec 2021 at 13:56, 陳智弘  wrote:
>
> +1
>
> Sijie Guo  於 2021年12月17日 週五 12:38 寫道:
>
> +1
>
> On Tue, Dec 14, 2021 at 11:20 AM Matteo Merli  wrote:
>
> https://github.com/apache/pulsar/issues/13306
>
>
> Pasted below for quoting convenience.
>
>
> 
>
> ## Motivation
>
> In Pulsar 2.8, we have introduced a setting to control the amount of
> memory
> used by a client instance.
>
> ```java
> interface ClientBuilder {
> ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> }
> ```
>
> By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> is
> being
> enforced.
>
> I think it's a good time for 2.10 to enable this setting by default
> and,
> correspondingly, to disable by default the producer queue size limit.
>
> This will simplify a lot the configuration that a producer application
> will
> have to come up with, when publishing with many topic/partitions or
> when messages
> are bigger than expected.
>
> ## Proposed changes
>
> In 2.10 release, for the `ClientBuilder`, change
> * `memoryLimit`: 0 -> 64 MB
>
> For the `ProducerBuilder`, changes
> * `maxPendingMessages`: 1000 -> 0
>
> 64MB is picked because it's a small enough memory size that will
> guarantee
> a very high producer throughput, irrespective of the individual
> messages
> size.
>
>
>
> --
> Matteo Merli
> 
>
>
>


Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

2021-12-20 Thread Michael Marshall
+1 sounds like a great improvement.

> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.

I haven't been able to follow the PIP 45 work, so I have some high
level questions. Do we have documentation on the implementation of
these re-acquirable resource locks? I'd like to understand how these
work. Also, are you able to describe the thorough testing? Were these
tests manual or are they already part of our GitHub CI or both?

Thanks,
Michael

On Tue, Dec 14, 2021 at 12:03 PM Matteo Merli  wrote:
>
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> ---
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that 
> allows
> brokers to continue operating in a safe mode when the session with ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane 
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> 


Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

2021-12-20 Thread Michael Marshall
+1

On Thu, Dec 16, 2021 at 10:37 PM Sijie Guo  wrote:
>
> +1
>
> On Tue, Dec 14, 2021 at 10:15 AM Matteo Merli  wrote:
>
> > Pasted below for quoting convenience.
> >
> >
> > 
> >
> > ## Motivation
> >
> > The consistent hashing implementation to uniformly assign keys to consumers
> > in the context of a KeyShared subscription, was introduced in
> > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> > 2.6.0.
> >
> > While consistent hashing can use slightly more memory in certain cases, it
> > is
> > more suitable as a general default implementation, as it leads to a fairer
> > distribution of keys across consumers, and avoiding corner cases that
> > depend
> > on the sequence of addition/removal of consumers.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > splitting ranges or
> > # consistent hashing to reassign keys to new consumers
> > subscriptionKeySharedUseConsistentHashing=false
> > ```
> >
> > Change its default value to `true`.
> >
> > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> > be
> > able to use the old implementation.
> >
> >
> >
> > --
> > Matteo Merli
> > 
> >


Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

2021-12-20 Thread mattison chao
+1

On Tue, 21 Dec 2021 at 06:30, Michael Marshall  wrote:

> +1
>
> On Thu, Dec 16, 2021 at 10:37 PM Sijie Guo  wrote:
> >
> > +1
> >
> > On Tue, Dec 14, 2021 at 10:15 AM Matteo Merli  wrote:
> >
> > > Pasted below for quoting convenience.
> > >
> > >
> > > 
> > >
> > > ## Motivation
> > >
> > > The consistent hashing implementation to uniformly assign keys to
> consumers
> > > in the context of a KeyShared subscription, was introduced in
> > > https://github.com/apache/pulsar/pull/6791, which was released in
> Pulsar
> > > 2.6.0.
> > >
> > > While consistent hashing can use slightly more memory in certain
> cases, it
> > > is
> > > more suitable as a general default implementation, as it leads to a
> fairer
> > > distribution of keys across consumers, and avoiding corner cases that
> > > depend
> > > on the sequence of addition/removal of consumers.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the setting:
> > >
> > > ```properties
> > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > splitting ranges or
> > > # consistent hashing to reassign keys to new consumers
> > > subscriptionKeySharedUseConsistentHashing=false
> > > ```
> > >
> > > Change its default value to `true`.
> > >
> > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will
> still
> > > be
> > > able to use the old implementation.
> > >
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
>


[GitHub] [pulsar-client-node] Matt-Esch commented on issue #99: Authenticating using a Token

2021-12-20 Thread GitBox


Matt-Esch commented on issue #99:
URL: 
https://github.com/apache/pulsar-client-node/issues/99#issuecomment-998386200


   Can confirm this is an issue, occurs when upgrading to pulsar > 2.6.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




Re: [VOTE] Pulsar Release 2.7.4 Candidate 1

2021-12-20 Thread guo jiwei
Hi, community.
Due to the CVE-2021-25105 (more details at
https://logging.apache.org/log4j/2.x/security.html),  we have to upgrade
log4j2 to 2.17,  so suggest canceling this vote.
I will send out the second-round vote later.

Regards
Jiwei Guo (Tboy)


On Mon, Dec 20, 2021 at 6:39 PM Nozomi Kurihara  wrote:

> +1 (binding)
>
> - verified checksums and signatures
> - build from source
> - verified produce/consume and functions
> - verified cassandra connector
> - verified stateful function
>
> Thanks,
> Nozomi
>
> 2021年12月18日(土) 19:43 Enrico Olivelli :
>
> > Jiwei Guo,
> > Thanks for driving the release
> > today is Saturday, so I am not sure how many people will have time to
> test
> > the release candidate during the 72 hours period (for instance I can do
> it
> > only on Monday, hopefully).
> > Please take this into consideration when you are going to close the VOTE,
> > maybe waiting until Tuesday may be a good idea.
> >
> > Enrico
> >
> >
> >
> > Il giorno sab 18 dic 2021 alle ore 09:54 guo jiwei  >
> > ha scritto:
> >
> > > This is the first release candidate for Apache Pulsar, version 2.7.4.
> > >
> > > It fixes the following issues:
> > >
> > >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
> > >
> > > Release note:
> > > https://github.com/apache/pulsar/pull/13391
> > >
> > > *** Please download, test and vote on this release. This vote will
> > > stay openfor at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> for
> > > convenience.
> > >
> > > Source and binary files:
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-1/
> > >
> > > SHA-512 checksums:
> > >
> > >
> > >
> >
> 1a34a408085328500c8c4c72e5ca9b458058a088674b1bfd550fe01ad365182b8820516f99b5c010a03037e1886ce5539cb57f5ef4a2297f1b7ed8fb5844bcd0
> > >  apache-pulsar-2.7.4-bin.tar.gz
> > >
> > >
> >
> 37e75698946f3390a254fa2bd7d974fca27d1aed632e86d64a08dc0bf8f2e0ca830cbeed2820782e59ac6bac7368ae69b54f1f5cd3d6c2cad292166e3538e285
> > >  apache-pulsar-2.7.4-src.tar.gz
> > >
> > >
> > > Maven staging repo:
> > >
> https://repository.apache.org/content/repositories/orgapachepulsar-1113
> > >
> > > The tag to be voted upon:
> > > v2.7.4-candidate-1 (ab451b855d873a9bad2005f939a23118a583baa9)
> > > https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-1
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download the source package, and follow the README to build
> > > and run the Pulsar standalone service.
> > >
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
> > >
> >
>


Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

2021-12-20 Thread Lin Lin



On 2021/12/14 18:03:20 Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/13304
> 
> 
> Pasted below for quoting convenience.
> 
> ---
> 
> 
> ## Motivation
> 
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
> 
> Another important change was to avoid doing any deferrable metadata operation
> when we know that we are not currently connected to the metadata service.
> 
> Finally, that enabled stabilization in 2.9 the configuration setting that 
> allows
> brokers to continue operating in a safe mode when the session with ZooKeeper
> expires.
> 
> The way it works is that, when we lose a ZooKeeper session, the data plane 
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid any
> inconsistencies.
> 
> New topics are not able to get started, but existing topics will see no
> impact.
> 
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
> 
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
> 
> ## Proposed changes
> 
> In 2.10 release, for the setting:
> 
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
> 
> Change its default value to `reconnect`.
> 
> 
> --
> Matteo Merli
> 
> 

+1

Please pay attention to the following scenarios:
I tried to use `Reconnect` in the production environment, and there was a 
serious online failure: two Brokers kept opening and closing the Ledger of the 
same topic, and the Ledger of BK kept throwing fencing exceptions.


Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-20 Thread Hiroyuki Sakai
+1 (binding)

 - check signatures/checksums
 - Built sources
 - Validate Pub/Sub and Java Functions
 - Validate Connectors
 - Validate Stateful Functions
​
Regards,
Hiroyuki



From: linlin 
Sent: Monday, December 20, 2021 20:18
To: dev@pulsar.apache.org 
Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2

This is the second release candidate for Apache Pulsar, version 2.8.2

It fixes the following issues:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rEmdtatsDR3P9Vw2mIMkfagjE8nQyAvb4C2k7ByuTMs%3D&reserved=0

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.
Source and binary files:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PBXIUy6WYNWmSb%2F7cJM1C6mbFlfsRnnrnfhHvoMj9ag%3D&reserved=0

SHA-512 checksums:
59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
 apache-pulsar-2.8.2-bin.tar.gz

82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
 apache-pulsar-2.8.2-src.tar.gz

Maven staging repo:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=98ZrSwMUZgKSrqA%2F9GGVCcuYEKIokc0bhaMPAgy9fDs%3D&reserved=0

The tag to be voted upon:
v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Y2cOvtt%2BmSwoQM2Z7ZWX31MwjseFCl9Y93FxYgyN2Iw%3D&reserved=0

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=VyKYHq2dz1jUrJruX6jfgXh1378poKCBFDe2K24HvGs%3D&reserved=0

Release notes draft:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F13400&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C19af5d4d0d5740e89cfb08d9c3aa8d32%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637755959550455483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=GGOM5mVzOaZfoVZawgwP%2BxRslSkJjTrLS2rSK6ma7qM%3D&reserved=0

Please download the source package, and follow the README to build
and run the Pulsar standalone service.


Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

2021-12-20 Thread PengHui Li
Hi Lin Lin

Which version you are working on the test?
Do you have any steps to reproduce the issue?
We'd better fix the problem in 2.10.

Regards,
Penghui

On Tue, Dec 21, 2021 at 11:06 AM Lin Lin  wrote:

>
>
> On 2021/12/14 18:03:20 Matteo Merli wrote:
> > https://github.com/apache/pulsar/issues/13304
> >
> >
> > Pasted below for quoting convenience.
> >
> > ---
> >
> >
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and
> 2.9
> > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled stabilization in 2.9 the configuration setting
> that allows
> > brokers to continue operating in a safe mode when the session with
> ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data
> plane will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that
> we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when a broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Change its default value to `reconnect`.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>
> +1
>
> Please pay attention to the following scenarios:
> I tried to use `Reconnect` in the production environment, and there was a
> serious online failure: two Brokers kept opening and closing the Ledger of
> the same topic, and the Ledger of BK kept throwing fencing exceptions.
>