date:20211221

OSX packages status....broken ?!?

2021-12-21 Thread Enrico Olivelli

Hello,
According to the release procedure I am trying to build the OSX packages
but the build fails.

This is the issue:
https://github.com/apache/pulsar/issues/13429

Apart from fixing the problem itself, I wonder if it would be better to
create a CI job that periodically runs such a procedure.

The net result is:
- the procedure does not work
- even if it worked, there is no guarantee that the packages work well

In the meantime I will proceed to the next step in the release process for
2.9.1

Thoughts ?


Enrico

PR merge

2021-12-21 Thread zhangao

Hi, Please take a look at this two issue, it has been for a lone time. I think 
it's ready to merge. 
https://github.com/apache/pulsar/pull/11893
https://github.com/apache/pulsar/pull/12025

RE: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-21 Thread Masahiro Sakamoto

+1 (binding)

- Checked checksums and signatures
- Checked license headers using Apache Rat
- Compiled the source
- Ran the standalone server
- Confirmed that producer and consumer work properly
- Validated functions, connectors, and stateful functions

Regards,

Masahiro Sakamoto
Yahoo Japan Corp.
E-mail: massa...@yahoo-corp.jp

-Original Message-
From: Hiroyuki Sakai  
Sent: Tuesday, December 21, 2021 1:07 PM
To: dev@pulsar.apache.org
Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

+1 (binding)

 - check signatures/checksums
 - Built sources
 - Validate Pub/Sub and Java Functions
 - Validate Connectors
 - Validate Stateful Functions

Regards,
Hiroyuki



From: linlin 
Sent: Monday, December 20, 2021 20:18
To: dev@pulsar.apache.org 
Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2

This is the second release candidate for Apache Pulsar, version 2.8.2

It fixes the following issues:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.
Source and binary files:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0

SHA-512 checksums:
59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
 apache-pulsar-2.8.2-bin.tar.gz

82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
 apache-pulsar-2.8.2-src.tar.gz

Maven staging repo:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4TVSZI76N8xalmLLwJyvtUJkkpNC7T0s2Taj%2Fc5CKgg%3D&reserved=0

The tag to be voted upon:
v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Ba6OsrGNvcCsuYFgPp%2Btg49yRHK%2FVBmWumN7haK%2Birw%3D&reserved=0

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EyKfkM0wHvCOASccnZw4QN6yjtSp8lzDrKAO%2FnB3IXk%3D&reserved=0

Release notes draft:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F13400&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l1tCWGIinNawL%2B%2Bui0PwSLlBn0UmKT5nX8dmH1Wfidg%3D&reserved=0

Please download the source package, and follow the README to build
and run the Pulsar standalone service.

[VOTE] Pulsar Release 2.7.4 Candidate 2

2021-12-21 Thread guo jiwei

This is the second release candidate for Apache Pulsar, version 2.7.4.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+

Release note:
https://github.com/apache/pulsar/pull/13391

*** Please download, test and vote on this release. This vote will
stay openfor at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-2/


SHA-512 checksums:

a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
 apache-pulsar-2.7.4-bin.tar.gz
03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
 apache-pulsar-2.7.4-src.tar.gz


Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1
130

The tag to be voted upon:
v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-
2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.



Regards
Jiwei Guo (Tboy)

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

2021-12-21 Thread Nozomi Kurihara

+1 (binding)

- Verified checksums and signatures
- Verified license headers by apache rat
- Built the source
- Ran standalone, producer, and consumer
- Verified functions, cassandra connectors and stateful functions

Thanks for driving the release.
Nozomi

2021年12月21日(火) 18:34 guo jiwei :

> This is the second release candidate for Apache Pulsar, version 2.7.4.
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
>
> Release note:
> https://github.com/apache/pulsar/pull/13391
>
> *** Please download, test and vote on this release. This vote will
> stay openfor at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-2/
> 
>
> SHA-512 checksums:
>
>
> a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
>  apache-pulsar-2.7.4-bin.tar.gz
>
> 03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
>  apache-pulsar-2.7.4-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1
>  >130
>
> The tag to be voted upon:
> v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
> https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-
> 2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
>
>
> Regards
> Jiwei Guo (Tboy)
>

Re: [DISCUSS] Release Pulsar 2.7.4

2021-12-21 Thread Shivji Kumar Jha

Hi Pulsar Team,

2.16.0 is prone to DDoS attacks [1].  Is it possible to move to 2.17.0 in
pulsar 2.7.4 ?

[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105

Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512


On Tue, 21 Dec 2021 at 02:46, Michael Marshall  wrote:

> Thank you Penghui, Technoboy, and BewareMyPower for picking up where I
> left off to get the 2.7.4 release ready to go. You cherry-picked many
> commits,
> resolved many conflicts, and fixed a bunch of tests!
>
> Thanks,
> Michael
>
>
> On Thu, Dec 16, 2021 at 7:37 PM guo jiwei  wrote:
> >
> > Hi,
> > After we have fixed some issue like ZookeeperCache NPE, listing namespace
> > exception, and skip some flaky tests (verified locally), now the CI have
> > passed.
> > Skipped flaky tests are tracked here:
> > https://github.com/apache/pulsar/issues/13299
> > Now we decide to vote for releasing 2.7.4.
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> >
> > On Tue, Dec 14, 2021 at 11:58 AM PengHui Li  wrote:
> >
> > > Thanks for the update, I will move it 2.7.5
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Tue, Dec 14, 2021 at 9:47 AM Matteo Merli 
> > > wrote:
> > >
> > > > Let's take https://github.com/apache/pulsar/pull/12484 out of the
> > > > picture since it's failing the tests.
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
> > > > On Sun, Dec 12, 2021 at 11:06 PM PengHui Li 
> wrote:
> > > > >
> > > > > Yes,
> > > > >
> > > > > https://github.com/apache/pulsar/pull/13215 has cherry-picked, so
> we
> > > can
> > > > > close it.
> > > > > https://github.com/apache/pulsar/pull/12484 blocked by the test.
> > > > >
> > > > > Penghui
> > > > >
> > > > > On Mon, Dec 13, 2021 at 2:35 PM Dave Fisher  >
> > > > wrote:
> > > > >
> > > > > > I see 2 PRs still open at
> > > > > >
> > > >
> > >
> https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.7.4
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Dec 12, 2021, at 8:22 PM, guo jiwei 
> > > wrote:
> > > > > > >
> > > > > > > I have pushed out some fixes in
> > > > > > https://github.com/apache/pulsar/pull/13243
> > > > > > > After the tests get passed, I will send out the RC-1 VOTE for
> 2.7.4
> > > > > > >
> > > > > > > Regards
> > > > > > > Jiwei Guo (Tboy)
> > > > > > >
> > > > > > >
> > > > > > >> On Sun, Dec 12, 2021 at 3:11 PM PengHui Li <
> peng...@apache.org>
> > > > wrote:
> > > > > > >>
> > > > > > >> Just put an update here. We have done the PR cherry-picking
> > > > > > >>
> > > > > > >> https://github.com/apache/pulsar/commits/branch-2.7
> > > > > > >>
> > > > > > >> And most of the integration tests are fixed due to the docker
> > > image
> > > > > > issue
> > > > > > >> or the testcontainer issue, now some integration tests get
> passed,
> > > > but
> > > > > > some
> > > > > > >> are not.
> > > > > > >> And there are some failed tests, maybe a flaky test, we need
> to
> > > > ensure
> > > > > > it's
> > > > > > >> not a regression.
> > > > > > >>
> > > > > > >> We are continuing on the test part.
> > > > > > >>
> > > > > > >> Penghui
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>> On Sat, Dec 11, 2021 at 5:36 PM PengHui Li <
> peng...@apache.org>
> > > > wrote:
> > > > > > >>>
> > > > > > >>> Hi Michael,
> > > > > > >>>
> > > > > > >>> +1,
> > > > > > >>>
> > > > > > >>> Thanks for the great work.
> > > > > > >>> We will continue on the PR cherry-picking and the release
> process
> > > > to
> > > > > > make
> > > > > > >>> sure the urgent release can be done ASAP.
> > > > > > >>>
> > > > > > >>> Penghui
> > > > > > >>>
> > > > > > >>> On Sat, Dec 11, 2021 at 3:42 PM Michael Marshall <
> > > > mmarsh...@apache.org
> > > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > >  Given the log4j CVE, we should work to release 2.7.4.
> > > > > > 
> > > > > >  I started preparing the release today by cherry-picking
> merged
> > > PRs
> > > > > >  that have the `release/2.7.4` label but have not yet been
> > > > > >  cherry-picked to `branch-2.7` [0]. There are still 37 PRs
> that
> > > > have
> > > > > >  not been cherry picked. I think it will take too long to
> cherry
> > > > pick
> > > > > >  all of these commits, as many have conflicts, and we should
> > > > prioritize
> > > > > >  releasing 2.7.4. The main commits that we should get
> > > cherry-picked
> > > > > >  before creating the git tag are any labeled with
> > > > `component/security`.
> > > > > >  There are only a few remaining commits to cherry pick.
> Please
> > > let
> > > > me
> > > > > >  know if you think any other commits ought to be
> cherry-picked.
> > > > > > 
> > > > > >  The earliest I'll be able to build the release is Monday.
> If we
> > > > need
> > > > > >  to start sooner, perhaps someone else will be available to
> > > manage
> > > > this
> > > > > >  urgent release.
> > > > > > 
> > > > > >  Thanks,
> > > > > >  Michael
> > > > >

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-21 Thread Shivji Kumar Jha

Hi LinLin,

Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we move
to Log4j 2.17.0 in 2.8.2?

Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did not
> protect from uncontrolled recursion from self-referential lookups. This
> allows an attacker with control over Thread Context Map data to cause a
> denial of service when a crafted string is interpreted. This issue was
> fixed in Log4j 2.17.0 and 2.12.3.



[1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105

Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512


On Tue, 21 Dec 2021 at 14:42, Masahiro Sakamoto 
wrote:

> +1 (binding)
>
> - Checked checksums and signatures
> - Checked license headers using Apache Rat
> - Compiled the source
> - Ran the standalone server
> - Confirmed that producer and consumer work properly
> - Validated functions, connectors, and stateful functions
>
> Regards,
>
> Masahiro Sakamoto
> Yahoo Japan Corp.
> E-mail: massa...@yahoo-corp.jp
>
> -Original Message-
> From: Hiroyuki Sakai 
> Sent: Tuesday, December 21, 2021 1:07 PM
> To: dev@pulsar.apache.org
> Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2
>
> +1 (binding)
>
>  - check signatures/checksums
>  - Built sources
>  - Validate Pub/Sub and Java Functions
>  - Validate Connectors
>  - Validate Stateful Functions
> 
> Regards,
> Hiroyuki
>
>
> 
> From: linlin 
> Sent: Monday, December 20, 2021 20:18
> To: dev@pulsar.apache.org 
> Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2
>
> This is the second release candidate for Apache Pulsar, version 2.8.2
>
> It fixes the following issues:
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0
>
> *** Please download, test and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> Source and binary files:
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0
>
> SHA-512 checksums:
>
> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
>  apache-pulsar-2.8.2-bin.tar.gz
>
>
> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
>  apache-pulsar-2.8.2-src.tar.gz
>
> Maven staging repo:
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4TVSZI76N8xalmLLwJyvtUJkkpNC7T0s2Taj%2Fc5CKgg%3D&reserved=0
>
> The tag to be voted upon:
> v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Ba6OsrGNvcCsuYFgPp%2Btg49yRHK%2FVBmWumN7haK%2Birw%3D&reserved=0
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EyKfkM0wHvCOASccnZw4QN6yjtSp8lzDrKAO%2FnB3IXk%3D&reserved=0
>
> Release notes draft:
>
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F13400&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbG

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-21 Thread Enrico Olivelli

Il giorno mar 21 dic 2021 alle ore 11:49 Shivji Kumar Jha <
shiv4...@gmail.com> ha scritto:

> Hi LinLin,
>
> Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we move
> to Log4j 2.17.0 in 2.8.2?
>

is it included
https://github.com/apache/pulsar/tree/v2.8.2-candidate-2

Enrico


>
> Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did not
> > protect from uncontrolled recursion from self-referential lookups. This
> > allows an attacker with control over Thread Context Map data to cause a
> > denial of service when a crafted string is interpreted. This issue was
> > fixed in Log4j 2.17.0 and 2.12.3.
>
>
>
> [1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Tue, 21 Dec 2021 at 14:42, Masahiro Sakamoto 
> wrote:
>
> > +1 (binding)
> >
> > - Checked checksums and signatures
> > - Checked license headers using Apache Rat
> > - Compiled the source
> > - Ran the standalone server
> > - Confirmed that producer and consumer work properly
> > - Validated functions, connectors, and stateful functions
> >
> > Regards,
> >
> > Masahiro Sakamoto
> > Yahoo Japan Corp.
> > E-mail: massa...@yahoo-corp.jp
> >
> > -Original Message-
> > From: Hiroyuki Sakai 
> > Sent: Tuesday, December 21, 2021 1:07 PM
> > To: dev@pulsar.apache.org
> > Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2
> >
> > +1 (binding)
> >
> >  - check signatures/checksums
> >  - Built sources
> >  - Validate Pub/Sub and Java Functions
> >  - Validate Connectors
> >  - Validate Stateful Functions
> >
> > Regards,
> > Hiroyuki
> >
> >
> > 
> > From: linlin 
> > Sent: Monday, December 20, 2021 20:18
> > To: dev@pulsar.apache.org 
> > Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2
> >
> > This is the second release candidate for Apache Pulsar, version 2.8.2
> >
> > It fixes the following issues:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> > Source and binary files:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0
> >
> > SHA-512 checksums:
> >
> >
> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
> >  apache-pulsar-2.8.2-bin.tar.gz
> >
> >
> >
> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
> >  apache-pulsar-2.8.2-src.tar.gz
> >
> > Maven staging repo:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4TVSZI76N8xalmLLwJyvtUJkkpNC7T0s2Taj%2Fc5CKgg%3D&reserved=0
> >
> > The tag to be voted upon:
> > v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Ba6OsrGNvcCsuYFgPp%2Btg49yRHK%2FVBmWumN7haK%2Birw%3D&reserved=0
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

2021-12-21 Thread Zike Yang

Thanks for your review Michael.

On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall  wrote:
>
> Thanks for creating this PIP Zike Yang. This sounds like an important
> improvement. I have a couple thoughts.
>
> 1. Does the `retryLetterTopic` have the same problem with immediate
> message expiration? Based on a quick glance, I don't see anything
> creating a subscription for the retry topic.
>
Yes. I think we also need to handle the retryLetterTopic. I will update the PIP.

> 2. Your prototype exposes a lacking in our existing client
> implementation: the non-admin client cannot directly create
> subscriptions. IIUC, this implementation does not work if
> `allowAutoSubscriptionCreation` is false.
IMO, we won't create the init subscription if `allowAutoSubscriptionCreation`
 is false. Otherwise, it will confuse the user. Users can explicitly create that
subscription to handle this case.

> Further, it is inefficient
> to create and immediately close a consumer just to create a
> subscription. Perhaps we should just update the non-admin client so
> that it can create subscriptions? It could be private so that we don't
> create confusion with the non-admin and the admin clients.
Do you mean to create a private method called `createSubscription` in
the non-admin client? But we don't have a protocol command for
creating subscription.
So in that method, we can send the `CommandSubscribe` and `CommandUnsubscribe`
to the broker to simulate the creation and closure of a consumer. But
I think it is not
straightforward to create a subscription using two commands.
Or the other way is to create a new command `CommandSubscription` in
the protocol
to handle this. But I wonder if there are more use cases for this
command that would
make it worthwhile to use this solution.
What do you think about these two ways?

>
> Thanks,
> Michael
>
>
> On Mon, Dec 20, 2021 at 4:18 AM Zike Yang
>  wrote:
> >
> > https://github.com/apache/pulsar/issues/13408
> >
> > Pasted below for quoting convenience.
> >
> > ——
> >
> >
> > ## Motivation
> >
> > If we enable the DLQ when consuming messages. For some messages that can't 
> > be processed successfully, the messages will be moved to the DLQ, but if we 
> > do not specify the data retention for the namespace or create a 
> > subscription for the DLQ to retain the data, the data of the DLQ topic will 
> > be removed automatically. Therefore, we need to create the initial 
> > subscription before sending messages to the DLQ.
> >
> > ## Goal
> >
> > Users can set the initial subscription name in the DeadLetterPolicy. The 
> > consumer will create the initial subscription before sending messages to 
> > the DLQ. At this point, subsequent messages produced to the DLQ are not 
> > automatically deleted unexpectedly.
> >
> > This PIP needs to be compatible with the previous behavior. The initial 
> > subscription name in the DeadLetterPolicy is optional. Default value is the 
> > subscription name of the consumer.
> >
> > ## API Changes
> >
> > Add `initSubscriptionName` to the `DeadLetterPolicy`
> >
> > ```java
> > /**
> >  * Name of the initial subscription name of the dead letter topic.
> >  * The default value is the subscription name of the consumer.
> >  */
> > private String initSubscriptionName;
> > ```
> >
> > ## Implementation
> >
> > Before the deadLetterProducer is initialized, the consumer first tries to 
> > create a deadLetterConsumer using the initial subscription name in the 
> > DeadLetterPolicy. When this subscription already exists, the ConsumerBusy 
> > exception will occur. In this case, we can ignore that exception and create 
> > the deadLetterProducer.
> >
> > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
> >
> >
> >
> > Thanks,
> > Zike Yang
> >



Best regards,
Zike Yang

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

2021-12-21 Thread Enrico Olivelli

+1 (binding)

- built from sources, JDK8 in MacOS
- run pulsar standalone, smoke tests
- verified checksum and digital signatures
- I took a look to the Maven staging repository

Thanks for driving the release

Enrico


Il giorno mar 21 dic 2021 alle ore 11:27 Nozomi Kurihara <
nkuri...@apache.org> ha scritto:

> +1 (binding)
>
> - Verified checksums and signatures
> - Verified license headers by apache rat
> - Built the source
> - Ran standalone, producer, and consumer
> - Verified functions, cassandra connectors and stateful functions
>
> Thanks for driving the release.
> Nozomi
>
> 2021年12月21日(火) 18:34 guo jiwei :
>
> > This is the second release candidate for Apache Pulsar, version 2.7.4.
> >
> > It fixes the following issues:
> >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
> >
> > Release note:
> > https://github.com/apache/pulsar/pull/13391
> >
> > *** Please download, test and vote on this release. This vote will
> > stay openfor at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-2/
> >  >
> >
> > SHA-512 checksums:
> >
> >
> >
> a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
> >  apache-pulsar-2.7.4-bin.tar.gz
> >
> >
> 03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
> >  apache-pulsar-2.7.4-src.tar.gz
> >
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1
> >  > >130
> >
> > The tag to be voted upon:
> > v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
> > https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-
> > 2
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
>

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-21 Thread Enrico Olivelli

+1 (binding)

- built from sources, JDK8 in MacOS
- run pulsar standalone, smoke tests
- verified checksum and digital signatures
- I took a look to the Maven staging repository (verified that "integration
tests jars" are present)

Thanks for driving the release

Enrico

Il giorno mar 21 dic 2021 alle ore 11:54 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

>
>
> Il giorno mar 21 dic 2021 alle ore 11:49 Shivji Kumar Jha <
> shiv4...@gmail.com> ha scritto:
>
>> Hi LinLin,
>>
>> Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we
>> move
>> to Log4j 2.17.0 in 2.8.2?
>>
>
> is it included
> https://github.com/apache/pulsar/tree/v2.8.2-candidate-2
>
> Enrico
>
>
>>
>> Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did
>> not
>> > protect from uncontrolled recursion from self-referential lookups. This
>> > allows an attacker with control over Thread Context Map data to cause a
>> > denial of service when a crafted string is interpreted. This issue was
>> > fixed in Log4j 2.17.0 and 2.12.3.
>>
>>
>>
>> [1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
>>
>> Regards,
>> Shivji Kumar Jha
>> http://www.shivjijha.com/
>> +91 8884075512
>>
>>
>> On Tue, 21 Dec 2021 at 14:42, Masahiro Sakamoto 
>> wrote:
>>
>> > +1 (binding)
>> >
>> > - Checked checksums and signatures
>> > - Checked license headers using Apache Rat
>> > - Compiled the source
>> > - Ran the standalone server
>> > - Confirmed that producer and consumer work properly
>> > - Validated functions, connectors, and stateful functions
>> >
>> > Regards,
>> >
>> > Masahiro Sakamoto
>> > Yahoo Japan Corp.
>> > E-mail: massa...@yahoo-corp.jp
>> >
>> > -Original Message-
>> > From: Hiroyuki Sakai 
>> > Sent: Tuesday, December 21, 2021 1:07 PM
>> > To: dev@pulsar.apache.org
>> > Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2
>> >
>> > +1 (binding)
>> >
>> >  - check signatures/checksums
>> >  - Built sources
>> >  - Validate Pub/Sub and Java Functions
>> >  - Validate Connectors
>> >  - Validate Stateful Functions
>> >
>> > Regards,
>> > Hiroyuki
>> >
>> >
>> > 
>> > From: linlin 
>> > Sent: Monday, December 20, 2021 20:18
>> > To: dev@pulsar.apache.org 
>> > Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2
>> >
>> > This is the second release candidate for Apache Pulsar, version 2.8.2
>> >
>> > It fixes the following issues:
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0
>> >
>> > *** Please download, test and vote on this release. This vote will stay
>> > open
>> > for at least 72 hours ***
>> >
>> > Note that we are voting upon the source (tag), binaries are provided for
>> > convenience.
>> > Source and binary files:
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0
>> >
>> > SHA-512 checksums:
>> >
>> >
>> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
>> >  apache-pulsar-2.8.2-bin.tar.gz
>> >
>> >
>> >
>> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
>> >  apache-pulsar-2.8.2-src.tar.gz
>> >
>> > Maven staging repo:
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4TVSZI76N8xalmLLwJyvtUJkkpNC7T0s2Taj%2Fc5CKgg%3D&reserved=0
>> >
>> > The tag to be voted upon:
>> > v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

2021-12-21 Thread Michael Marshall

> IMO, we won't create the init subscription if `allowAutoSubscriptionCreation`
> is false. Otherwise, it will confuse the user. Users can explicitly create 
> that
> subscription to handle this case.

That is a good point, but I don't think we should attempt to create
the subscription implicitly via consumer creation because that will
lead to additional failure, which adds unnecessary load on the broker.

We can avoid confusion by only attempting to create the subscription
when `initSubscriptionName` is not `null`. As a result, this PIP
wouldn't change the behavior for current implementations.

> Do you mean to create a private method called `createSubscription` in
> the non-admin client? But we don't have a protocol command for
> creating subscription

Yes, that was what I meant, and you're right, we don't have a protocol
command for this case.

> Or the other way is to create a new command `CommandSubscription` in
> the protocol to handle this.

+1 - I think we should expand the binary protocol to add a
`CommandSubscription` or possibly `CommandCreateSubscription`
(`CommandSubscribe` is already taken). Without a new command, we will
introduce race conditions related to consumer creation and will
definitely generate unnecessary errors.

Regarding default authorization, this new command would require the
same level of permission for the client since the act of creating a
consumer and the act of creating a subscription both require the same
`AuthAction` permission: Consume.

Thanks,
Michael

On Tue, Dec 21, 2021 at 4:57 AM Zike Yang

 wrote:
>
> Thanks for your review Michael.
>
> On Tue, Dec 21, 2021 at 5:48 AM Michael Marshall  wrote:
> >
> > Thanks for creating this PIP Zike Yang. This sounds like an important
> > improvement. I have a couple thoughts.
> >
> > 1. Does the `retryLetterTopic` have the same problem with immediate
> > message expiration? Based on a quick glance, I don't see anything
> > creating a subscription for the retry topic.
> >
> Yes. I think we also need to handle the retryLetterTopic. I will update the 
> PIP.
>
> > 2. Your prototype exposes a lacking in our existing client
> > implementation: the non-admin client cannot directly create
> > subscriptions. IIUC, this implementation does not work if
> > `allowAutoSubscriptionCreation` is false.
> IMO, we won't create the init subscription if `allowAutoSubscriptionCreation`
>  is false. Otherwise, it will confuse the user. Users can explicitly create 
> that
> subscription to handle this case.
>
> > Further, it is inefficient
> > to create and immediately close a consumer just to create a
> > subscription. Perhaps we should just update the non-admin client so
> > that it can create subscriptions? It could be private so that we don't
> > create confusion with the non-admin and the admin clients.
> Do you mean to create a private method called `createSubscription` in
> the non-admin client? But we don't have a protocol command for
> creating subscription.
> So in that method, we can send the `CommandSubscribe` and `CommandUnsubscribe`
> to the broker to simulate the creation and closure of a consumer. But
> I think it is not
> straightforward to create a subscription using two commands.
> Or the other way is to create a new command `CommandSubscription` in
> the protocol
> to handle this. But I wonder if there are more use cases for this
> command that would
> make it worthwhile to use this solution.
> What do you think about these two ways?
>
> >
> > Thanks,
> > Michael
> >
> >
> > On Mon, Dec 20, 2021 at 4:18 AM Zike Yang
> >  wrote:
> > >
> > > https://github.com/apache/pulsar/issues/13408
> > >
> > > Pasted below for quoting convenience.
> > >
> > > ——
> > >
> > >
> > > ## Motivation
> > >
> > > If we enable the DLQ when consuming messages. For some messages that 
> > > can't be processed successfully, the messages will be moved to the DLQ, 
> > > but if we do not specify the data retention for the namespace or create a 
> > > subscription for the DLQ to retain the data, the data of the DLQ topic 
> > > will be removed automatically. Therefore, we need to create the initial 
> > > subscription before sending messages to the DLQ.
> > >
> > > ## Goal
> > >
> > > Users can set the initial subscription name in the DeadLetterPolicy. The 
> > > consumer will create the initial subscription before sending messages to 
> > > the DLQ. At this point, subsequent messages produced to the DLQ are not 
> > > automatically deleted unexpectedly.
> > >
> > > This PIP needs to be compatible with the previous behavior. The initial 
> > > subscription name in the DeadLetterPolicy is optional. Default value is 
> > > the subscription name of the consumer.
> > >
> > > ## API Changes
> > >
> > > Add `initSubscriptionName` to the `DeadLetterPolicy`
> > >
> > > ```java
> > > /**
> > >  * Name of the initial subscription name of the dead letter topic.
> > >  * The default value is the subscription name of the consumer.
> > >  */
> > > private String initSubs

Re: [PR] Pulsar non root docker image

2021-12-21 Thread Michael Marshall

All tests are now passing for this PR [0]. I built the docker image
and pushed it to my personal repository to simplify testing [1] for
anyone interested in verifying the changes.

I would like our docker image to be as close to immutable as possible.
As far as I can tell, here are the only reasons the image cannot be
immutable:

1. Pulsar configuration is read in only from configuration files in
`/pulsar/conf`. A non root user must be able to update these files to
have run with custom configuration.
2. The Pulsar function worker unpacks nar files to
`/pulsar/download/pulsar_functions` by default.
3. Pulsar tiered storage writes to `/pulsar` by default when using
filesystem storage.
4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
5. Pulsar-admin and functions write to `/pulsar/log` by default
(possibly other components too).

Note that even though bookkeepers and zookeepers write to
`/pulsar/data`, they should be writing to docker volumes, in which
case, the host's file system permissions are all that matter.

If we can remove any of the above reasons, we can decrease the
privilege in the docker image.

The PR has more detail. Please take a look.

Thanks,
Michael

[0] https://github.com/apache/pulsar/pull/13376
[1] michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9

On Fri, Dec 17, 2021 at 12:33 AM Michael Marshall  wrote:
>
> Hi Pulsar Community,
>
> I opened a PR to make our pulsar and pulsar-all docker images non root
> and OpenShift compliant [0]. As some may remember, we had issues with
> these changes before due to lack of testing. I plan to test thoroughly
> before we merge this PR, and it'd be great to have others test too. I
> published a build of my PR [1]. I also have an issue [2] tracking this
> work.
>
> Please take a look. I hope to make our 2.10 release a non root release!
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/13376
> [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT
> [2] https://github.com/apache/pulsar/issues/11269

Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

2021-12-21 Thread Matteo Merli

On Mon, Dec 20, 2021 at 2:24 PM Michael Marshall  wrote:
> I haven't been able to follow the PIP 45 work, so I have some high
> level questions. Do we have documentation on the implementation of
> these re-acquirable resource locks? I'd like to understand how these
> work.

There is admittedly not much high-level doc on the locks. The
implementation class is the best reference atm:
https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java

Similarly, for leader election implementation:
https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/LeaderElectionImpl.java

> Also, are you able to describe the thorough testing? Were these
> tests manual or are they already part of our GitHub CI or both?

It's a combination of those. In the unit tests, through fault
injections, we're reproducing all the possible states.

Re: [DISCUSSION] PIP-120: Enable client memory limit by default

2021-12-21 Thread Matteo Merli

> Did you consider determining the default limit by inspecting the JVM's 
> runtime?
> For example, in the broker, we set `maxMessagePublishBufferSizeInMB` to 1/2
> of direct memory, by default.
>
> I concede that this config is more complicated than a broker config since
> we're talking about a client application. However, it seems like applications
> with 1 GB memory would probably want a different setting than one with
> 32 GB memory, and a fractional value could make the client work better
> "out of the box."
>
> We cannot make everyone happy with a default value, so if this dynamic
> default is too complicated to implement, I'm good with 64 MB. :)

The issue is that we don't know what are the application assumptions
with respect to the memory. Eg; even if it's given 32GB to the JVM, it
might want to use all of it.

The other aspect is that after a point, increasing the memory size,
won't increase the throughput anymore, and it will become just a waste
of RAM.

[VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

2021-12-21 Thread Matteo Merli

https://github.com/apache/pulsar/issues/13304

Following the discussion, I have updated the proposal to also include
the deprecation and renaming of the config setting name to
`metadataSessionExpiredPolicy`.



---
## Motivation

After all the work done for PIP-45 that was already included in 2.8 and 2.9
releases, it enabled the concept of re-acquirable resource locks and leader
election.

Another important change was to avoid doing any deferrable metadata operation
when we know that we are not currently connected to the metadata service.

Finally, that enabled to stabilize in 2.9 the configuration setting that allows
brokers to continue operating in a safe mode when the session with ZooKeeper
expires.

The way it works is that, when we lose a ZooKeeper session, the data plane will
continue to work undisturbed, relying on the BookKeeper fencing to avoid any
inconsistencies.

New topics are not able to get started, but existing topics will see no
impact.

The original intention for shutting down the brokers was to ensure that we
would automatically go back to a consistent state, with respect to which
resources are "owned" in ZooKeeper by a given broker.

With the re-acquirable resource locks, that problem was solved and thoroughly
tested to be robust.

## Proposed changes

In 2.10 release, for the setting:

```properties
# There are two policies to apply when broker metadata session
expires: session expired happens, "shutdown" or "reconnect".
# With "shutdown", the broker will be restarted.
# With "reconnect", the broker will keep serving the topics, while
attempting to recreate a new session.
zookeeperSessionExpiredPolicy=shutdown
```

Deprecate the old setting and rename it to
`metadataSessionExpiredPolicy`, with default value set to `reconnect`.

[VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

2021-12-21 Thread Matteo Merli

This is the voting thread for PIP-119. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13305

---

## Motivation

The consistent hashing implementation to uniformly assign keys to consumers
in the context of a KeyShared subscription, was introduced in
https://github.com/apache/pulsar/pull/6791, which was released in Pulsar 2.6.0.

While consistent hashing can use slightly more memory in certain cases, it is
more suitable as a general default implementation, as it leads to a fairer
distribution of keys across consumers, and avoiding corner cases that depend
on the sequence of addition/removal of consumers.

## Proposed changes

In 2.10 release, for the setting:

```properties
# On KeyShared subscriptions, with default AUTO_SPLIT mode, use
splitting ranges or
# consistent hashing to reassign keys to new consumers
subscriptionKeySharedUseConsistentHashing=false
```

Change its default value to `true`.

The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still be
able to use the old implementation.

--
Matteo Merli

[VOTE] PIP-120: Enable client memory limit by default

2021-12-21 Thread Matteo Merli

This is the voting thread for PIP-120. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13306



## Motivation

In Pulsar 2.8, we have introduced a setting to control the amount of memory
used by a client instance.

```java
interface ClientBuilder {
ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
}
```

By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is being
enforced.

I think it's a good time for 2.10 to enable this setting by default and,
correspondingly, to disable by default the producer queue size limit.

This will simplify a lot the configuration that a producer application will
have to come up with, when publishing with many topic/partitions or
when messages
are bigger than expected.

## Proposed changes

In 2.10 release, for the `ClientBuilder`, change
  * `memoryLimit`: 0 -> 64 MB

For the `ProducerBuilder`, changes
  * `maxPendingMessages`: 1000 -> 0

64MB is picked because it's a small enough memory size that will guarantee
a very high producer throughput, irrespective of the individual messages size.


--
Matteo Merli

Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults

2021-12-21 Thread Matteo Merli

--
Matteo Merli


On Sun, Dec 19, 2021 at 9:42 PM Lan Liang  wrote:
>
> My two cents +1 , In my case, i sometimes need to run pulsar standalone in 
> some  server with limited resources.
> The other point i can think of is maybe schema storage also need to change 
> from using BK to  using local filesystem or some else.

Since we're anyway going to have an embedded bookie, I think we're
fine to continue using BK for schema storage.

[VOTE] PIP-117: Change Pulsar standalone defaults

2021-12-21 Thread Matteo Merli

This is the voting thread for PIP-117. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13302



## Motivation

Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where
all the components are started within the context of a single JVM process.

Users are using the standalone as a way to get quickly started with Pulsar or
in all the cases where it makes sense to have a single node deployment.

Right now, the standalone is starting by default with many components,
several of
which are quite complex, since they are designed to be deployed in a distributed
fashion.

## Goal

Simplify the components of Pulsar standalone to achieve:

 1. Reduce complexity
 2. Reduce startup time
 3. Reduce memory and CPU footprint of running standalone

## Proposed changes

The proposal here is to change some of the default implementations that are
used for the Pulsar standalone.

 1. **Metadata Store implementation** -->
  Change from ZooKeeper to RocksDB

 2. **Pulsar functions package backend** -->
  Change from using DistributedLog to using local filesystem, storing the
  jars directly in the data folder instead of uploading them into BK.

 3. **Pulsar functions state store implementation** -->
  Change the state store to be backed by a MetadataStore based backed,
  with the RocksDB implementation.

 4. **Table Service** -->
  Do not start BK table service by default

## Compatibility considerations

In order to avoid compatibility issues where users have existing Pulsar
standalone services that they want to upgrade without conflicts, we will
follow the principle of keeping the old defaults where there is existing
data on the disk.

We will add a file, serving the purpose as a flag, in the `data/standalone`
directory, for example `new-2.10-defaults`.

If the file is present, or if the data directory is completely missing, we
will adopt the new set of default configuration settings.

If the file is not there, we will continue to use existing defaults and we will
not break the upgrade operation.


--
Matteo Merli

[VOTE] PIP-123: Introduce Pulsar metadata CLI tool

2021-12-21 Thread Matteo Merli

This is the voting thread for PIP-123. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13346


-
## Motivation

For a very long time, we have included a CLI command to start the ZooKeeper
shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
of the ZooKeeper tool `zkCli.sh`.

This is useful in some cases to either verify the content of metadata or
to perform cleanup and modification tasks for which there is not an available
option through the Pulsar REST APIs.

While it's very useful, there are some drawbacks with the `zookeeper-shell`
as it is today:

 1. This is only a ZooKeeper tool (obviously). Since we are adding more
metadata backends, we should have a tool that works across all the
implementations and presents a single consistent interface.

 2. ZooKeeper shell is designed to be an interactive shell and it's not very
good when trying to do non-interactive scriptable operations.

 3. ZooKeeper is a bit clunky when using it and it requires the user to retype
paths many times. The commands are not very intuitive or documented.
It's not possible to update z-node with multi-lines content.

 4. We cannot easily add functionality or improvements into ZooKeeper shell,
since it belongs to a different project and the tool has been stagnating
for many years.

 5. In cases where the z-nodes content is binary (Protobuf) or compressed, there
is no easy way to inspect the content from the ZooKeeper shell.
Additionally, we can format and colorize JSON content to make it easier to
read.

 6. The paths used for metadata resources are also often using encodings that
make it more difficult to construct on the shell tool.

Part of what is described here is in the `pulsar-managed-ledger-admin` CLI tool,
though that is a Python script that requires additional dependencies that are
not typically installed, it only works with ZooKeeper, and it only targets
accessing the managed ledger metadata.

## Goal

Introduce a new Java CLI tool to access, inspect and modify metadata that solves
all the issues described above.

We would leave the `zookeeper-shell` command for now. In the future, once the
new tool is proven, we can consider removing the `zookeeper-shell` command.


## Proposed changes

Add a new command:
```bash
bin/pulsar metadata
```

with several subcommands:


 Get

Examples:
```bash
# General path get
$ pulsar metadata get /my-path


# Topic metadata
$ pulsar metadata get topic my-tenant/my-namespace/my-topic
{
  # Managed ledger metadata
}

# Namespace get
$ pulsar metadata get namespace my-tenant/my-namespace
{
  # Namespace metadata
}

$ pulsar metadata get ledger 12345
{
  # BK ledger metadata
}
```

 Delete

Examples:
```bash
# General path delete
$ pulsar metadata delete /my-path

# Topic metadata
$ pulsar metadata delete topic my-tenant/my-namespace/my-topic
```

 Scan

Examples:
```bash
$ pulsar metadata scan /my-path
/my-path
/my-path/1
/my-path/2
/my-path/3
/my-path/3/1

$ pulsar metadata scan --values /my-path
/my-path
{value}

/my-path/1
{value}

/my-path/2
{value}
```

 Shell

```bash
$ pulsar metadata shell
> get topic my-tenant/my-namespace/my-topic
{
  # Managed ledger metadata
}

> delete topic my-tenant/my-namespace/my-topic

> cd /my-path
> ls
1
2
3
> delete 1 # Delete keys with relative paths

```


--
Matteo Merli

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

2021-12-21 Thread mattison chao

+1

On Wed, 22 Dec 2021 at 07:59, Matteo Merli  wrote:

> This is the voting thread for PIP-123. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13346
>
>
> -
> ## Motivation
>
> For a very long time, we have included a CLI command to start the ZooKeeper
> shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> of the ZooKeeper tool `zkCli.sh`.
>
> This is useful in some cases to either verify the content of metadata or
> to perform cleanup and modification tasks for which there is not an
> available
> option through the Pulsar REST APIs.
>
> While it's very useful, there are some drawbacks with the `zookeeper-shell`
> as it is today:
>
>  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> metadata backends, we should have a tool that works across all the
> implementations and presents a single consistent interface.
>
>  2. ZooKeeper shell is designed to be an interactive shell and it's not
> very
> good when trying to do non-interactive scriptable operations.
>
>  3. ZooKeeper is a bit clunky when using it and it requires the user to
> retype
> paths many times. The commands are not very intuitive or documented.
> It's not possible to update z-node with multi-lines content.
>
>  4. We cannot easily add functionality or improvements into ZooKeeper
> shell,
> since it belongs to a different project and the tool has been
> stagnating
> for many years.
>
>  5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> there
> is no easy way to inspect the content from the ZooKeeper shell.
> Additionally, we can format and colorize JSON content to make it
> easier to
> read.
>
>  6. The paths used for metadata resources are also often using encodings
> that
> make it more difficult to construct on the shell tool.
>
> Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> tool,
> though that is a Python script that requires additional dependencies that
> are
> not typically installed, it only works with ZooKeeper, and it only targets
> accessing the managed ledger metadata.
>
> ## Goal
>
> Introduce a new Java CLI tool to access, inspect and modify metadata that
> solves
> all the issues described above.
>
> We would leave the `zookeeper-shell` command for now. In the future, once
> the
> new tool is proven, we can consider removing the `zookeeper-shell` command.
>
>
> ## Proposed changes
>
> Add a new command:
> ```bash
> bin/pulsar metadata
> ```
>
> with several subcommands:
>
>
>  Get
>
> Examples:
> ```bash
> # General path get
> $ pulsar metadata get /my-path
>
>
> # Topic metadata
> $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> {
>   # Managed ledger metadata
> }
>
> # Namespace get
> $ pulsar metadata get namespace my-tenant/my-namespace
> {
>   # Namespace metadata
> }
>
> $ pulsar metadata get ledger 12345
> {
>   # BK ledger metadata
> }
> ```
>
>  Delete
>
> Examples:
> ```bash
> # General path delete
> $ pulsar metadata delete /my-path
>
> # Topic metadata
> $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> ```
>
>  Scan
>
> Examples:
> ```bash
> $ pulsar metadata scan /my-path
> /my-path
> /my-path/1
> /my-path/2
> /my-path/3
> /my-path/3/1
>
> $ pulsar metadata scan --values /my-path
> /my-path
> {value}
>
> /my-path/1
> {value}
>
> /my-path/2
> {value}
> ```
>
>  Shell
>
> ```bash
> $ pulsar metadata shell
> > get topic my-tenant/my-namespace/my-topic
> {
>   # Managed ledger metadata
> }
>
> > delete topic my-tenant/my-namespace/my-topic
>
> > cd /my-path
> > ls
> 1
> 2
> 3
> > delete 1 # Delete keys with relative paths
>
> ```
>
>
> --
> Matteo Merli
> 
>

Re: [VOTE] PIP-120: Enable client memory limit by default

2021-12-21 Thread mattison chao

+1

On Wed, 22 Dec 2021 at 07:46, Matteo Merli  wrote:

> This is the voting thread for PIP-120. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13306
>
> 
>
> ## Motivation
>
> In Pulsar 2.8, we have introduced a setting to control the amount of memory
> used by a client instance.
>
> ```java
> interface ClientBuilder {
> ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> }
> ```
>
> By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> being
> enforced.
>
> I think it's a good time for 2.10 to enable this setting by default and,
> correspondingly, to disable by default the producer queue size limit.
>
> This will simplify a lot the configuration that a producer application will
> have to come up with, when publishing with many topic/partitions or
> when messages
> are bigger than expected.
>
> ## Proposed changes
>
> In 2.10 release, for the `ClientBuilder`, change
>   * `memoryLimit`: 0 -> 64 MB
>
> For the `ProducerBuilder`, changes
>   * `maxPendingMessages`: 1000 -> 0
>
> 64MB is picked because it's a small enough memory size that will guarantee
> a very high producer throughput, irrespective of the individual messages
> size.
>
>
> --
> Matteo Merli
> 
>

Re: [PR] Pulsar non root docker image

2021-12-21 Thread Haiting Jiang

> 1. Pulsar configuration is read in only from configuration files in
> `/pulsar/conf`. A non root user must be able to update these files to
> have run with custom configuration.

About the configurations, I also see similar require like this lately [0]. 
IMHO, update any configs without redeploy service is a useful feature.
I would like post a PIP for this later. 
Basic idea is like make all configs dynamic by default except 
`metadataStoreUrl` and all configs are stored under path "/admin/configuration" 
in metadata store.

[0] https://github.com/apache/pulsar/pull/13074

On 2021/12/21 20:16:44 Michael Marshall wrote:
> All tests are now passing for this PR [0]. I built the docker image
> and pushed it to my personal repository to simplify testing [1] for
> anyone interested in verifying the changes.
> 
> I would like our docker image to be as close to immutable as possible.
> As far as I can tell, here are the only reasons the image cannot be
> immutable:
> 
> 1. Pulsar configuration is read in only from configuration files in
> `/pulsar/conf`. A non root user must be able to update these files to
> have run with custom configuration.
> 2. The Pulsar function worker unpacks nar files to
> `/pulsar/download/pulsar_functions` by default.
> 3. Pulsar tiered storage writes to `/pulsar` by default when using
> filesystem storage.
> 4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
> 5. Pulsar-admin and functions write to `/pulsar/log` by default
> (possibly other components too).
> 
> Note that even though bookkeepers and zookeepers write to
> `/pulsar/data`, they should be writing to docker volumes, in which
> case, the host's file system permissions are all that matter.
> 
> If we can remove any of the above reasons, we can decrease the
> privilege in the docker image.
> 
> The PR has more detail. Please take a look.
> 
> Thanks,
> Michael
> 
> [0] https://github.com/apache/pulsar/pull/13376
> [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9
> 
> 
> On Fri, Dec 17, 2021 at 12:33 AM Michael Marshall  
> wrote:
> >
> > Hi Pulsar Community,
> >
> > I opened a PR to make our pulsar and pulsar-all docker images non root
> > and OpenShift compliant [0]. As some may remember, we had issues with
> > these changes before due to lack of testing. I plan to test thoroughly
> > before we merge this PR, and it'd be great to have others test too. I
> > published a build of my PR [1]. I also have an issue [2] tracking this
> > work.
> >
> > Please take a look. I hope to make our 2.10 release a non root release!
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/13376
> > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT
> > [2] https://github.com/apache/pulsar/issues/11269
>

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

2021-12-21 Thread Lin Lin




On 2021/12/21 10:48:41 Shivji Kumar Jha wrote:
> Hi LinLin,
> 
> Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we move
> to Log4j 2.17.0 in 2.8.2?
> 
> Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did not
> > protect from uncontrolled recursion from self-referential lookups. This
> > allows an attacker with control over Thread Context Map data to cause a
> > denial of service when a crafted string is interpreted. This issue was
> > fixed in Log4j 2.17.0 and 2.12.3.


Already included

WebSite build CI job does work - blocker for announcing release !

2021-12-21 Thread Enrico Olivelli

Hello,
the WebSite build CI job is stuck
https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true

This is probably the problem, it is due to "crowdin" that is timing out

File 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
SKIPPED
File 'master/website/i18n/en.json'/ - - OK
Done in 2452.38s.
+ yarn run crowdin-download
yarn run v1.22.5
warning package.json: No license field
$ crowdin --config ../crowdin.yaml download -b master
Error: The operation was canceled.


Can anyone help ?
we are stuck and if we do not update the website we cannot announce the
releases with the security fixes


Enrico

Re: [PR] Pulsar non root docker image

2021-12-21 Thread Enrico Olivelli

Michael,

I would like to add this item to the list
https://github.com/apache/pulsar/pull/10815

Basically if you are running as non root, you cannot add tools on demand,
so we need to add basic tools, like netstat/vim/unzip otherwise when
you have a problem you are trapped.

there are ways to inject tools, but they are very hard to execute for
people who is not very experienced

Enrico


Il giorno mer 22 dic 2021 alle ore 04:40 Haiting Jiang <
jianghait...@apache.org> ha scritto:

> > 1. Pulsar configuration is read in only from configuration files in
> > `/pulsar/conf`. A non root user must be able to update these files to
> > have run with custom configuration.
>
> About the configurations, I also see similar require like this lately [0].
> IMHO, update any configs without redeploy service is a useful feature.
> I would like post a PIP for this later.
> Basic idea is like make all configs dynamic by default except
> `metadataStoreUrl` and all configs are stored under path
> "/admin/configuration" in metadata store.
>
> [0] https://github.com/apache/pulsar/pull/13074
>
> On 2021/12/21 20:16:44 Michael Marshall wrote:
> > All tests are now passing for this PR [0]. I built the docker image
> > and pushed it to my personal repository to simplify testing [1] for
> > anyone interested in verifying the changes.
> >
> > I would like our docker image to be as close to immutable as possible.
> > As far as I can tell, here are the only reasons the image cannot be
> > immutable:
> >
> > 1. Pulsar configuration is read in only from configuration files in
> > `/pulsar/conf`. A non root user must be able to update these files to
> > have run with custom configuration.
> > 2. The Pulsar function worker unpacks nar files to
> > `/pulsar/download/pulsar_functions` by default.
> > 3. Pulsar tiered storage writes to `/pulsar` by default when using
> > filesystem storage.
> > 4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
> > 5. Pulsar-admin and functions write to `/pulsar/log` by default
> > (possibly other components too).
> >
> > Note that even though bookkeepers and zookeepers write to
> > `/pulsar/data`, they should be writing to docker volumes, in which
> > case, the host's file system permissions are all that matter.
> >
> > If we can remove any of the above reasons, we can decrease the
> > privilege in the docker image.
> >
> > The PR has more detail. Please take a look.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/13376
> > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9
> >
> >
> > On Fri, Dec 17, 2021 at 12:33 AM Michael Marshall 
> wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > I opened a PR to make our pulsar and pulsar-all docker images non root
> > > and OpenShift compliant [0]. As some may remember, we had issues with
> > > these changes before due to lack of testing. I plan to test thoroughly
> > > before we merge this PR, and it'd be great to have others test too. I
> > > published a build of my PR [1]. I also have an issue [2] tracking this
> > > work.
> > >
> > > Please take a look. I hope to make our 2.10 release a non root release!
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/13376
> > > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT
> > > [2] https://github.com/apache/pulsar/issues/11269
> >
>

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

2021-12-21 Thread Enrico Olivelli

very good

Enrico

Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
mattisonc...@gmail.com> ha scritto:

> +1
>
> On Wed, 22 Dec 2021 at 07:59, Matteo Merli  wrote:
>
> > This is the voting thread for PIP-123. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13346
> >
> >
> > -
> > ## Motivation
> >
> > For a very long time, we have included a CLI command to start the
> ZooKeeper
> > shell utility: `pulsar zookeeper-shell`, which is essentially a
> repackaging
> > of the ZooKeeper tool `zkCli.sh`.
> >
> > This is useful in some cases to either verify the content of metadata or
> > to perform cleanup and modification tasks for which there is not an
> > available
> > option through the Pulsar REST APIs.
> >
> > While it's very useful, there are some drawbacks with the
> `zookeeper-shell`
> > as it is today:
> >
> >  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > metadata backends, we should have a tool that works across all the
> > implementations and presents a single consistent interface.
> >
> >  2. ZooKeeper shell is designed to be an interactive shell and it's not
> > very
> > good when trying to do non-interactive scriptable operations.
> >
> >  3. ZooKeeper is a bit clunky when using it and it requires the user to
> > retype
> > paths many times. The commands are not very intuitive or documented.
> > It's not possible to update z-node with multi-lines content.
> >
> >  4. We cannot easily add functionality or improvements into ZooKeeper
> > shell,
> > since it belongs to a different project and the tool has been
> > stagnating
> > for many years.
> >
> >  5. In cases where the z-nodes content is binary (Protobuf) or
> compressed,
> > there
> > is no easy way to inspect the content from the ZooKeeper shell.
> > Additionally, we can format and colorize JSON content to make it
> > easier to
> > read.
> >
> >  6. The paths used for metadata resources are also often using encodings
> > that
> > make it more difficult to construct on the shell tool.
> >
> > Part of what is described here is in the `pulsar-managed-ledger-admin`
> CLI
> > tool,
> > though that is a Python script that requires additional dependencies that
> > are
> > not typically installed, it only works with ZooKeeper, and it only
> targets
> > accessing the managed ledger metadata.
> >
> > ## Goal
> >
> > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > solves
> > all the issues described above.
> >
> > We would leave the `zookeeper-shell` command for now. In the future, once
> > the
> > new tool is proven, we can consider removing the `zookeeper-shell`
> command.
> >
> >
> > ## Proposed changes
> >
> > Add a new command:
> > ```bash
> > bin/pulsar metadata
> > ```
> >
> > with several subcommands:
> >
> >
> >  Get
> >
> > Examples:
> > ```bash
> > # General path get
> > $ pulsar metadata get /my-path
> >
> >
> > # Topic metadata
> > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > # Namespace get
> > $ pulsar metadata get namespace my-tenant/my-namespace
> > {
> >   # Namespace metadata
> > }
> >
> > $ pulsar metadata get ledger 12345
> > {
> >   # BK ledger metadata
> > }
> > ```
> >
> >  Delete
> >
> > Examples:
> > ```bash
> > # General path delete
> > $ pulsar metadata delete /my-path
> >
> > # Topic metadata
> > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > ```
> >
> >  Scan
> >
> > Examples:
> > ```bash
> > $ pulsar metadata scan /my-path
> > /my-path
> > /my-path/1
> > /my-path/2
> > /my-path/3
> > /my-path/3/1
> >
> > $ pulsar metadata scan --values /my-path
> > /my-path
> > {value}
> >
> > /my-path/1
> > {value}
> >
> > /my-path/2
> > {value}
> > ```
> >
> >  Shell
> >
> > ```bash
> > $ pulsar metadata shell
> > > get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > > delete topic my-tenant/my-namespace/my-topic
> >
> > > cd /my-path
> > > ls
> > 1
> > 2
> > 3
> > > delete 1 # Delete keys with relative paths
> >
> > ```
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

[DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

2021-12-21 Thread Lari Hotari

Dear Pulsar community members,

PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
repositories" was created in April 2020. The repositories for
pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
ago [2].

I'd like to propose that we continue with the PIP-62 plan asap, before
Apache Pulsar 2.10.0 is released.

I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
waiting for the Trino Pulsar PR [3] being completed.

I am volunteering to making the changes as per the PIP-62 plan.
When can we proceed with the plan? Do we need an explicit vote on the
mailing list about this?

BR,

Lari

[1]
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
[2]
https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
[3] https://github.com/trinodb/trino/pull/8020

Other email threads:
* [Discuss] Don't include presto/trino in the normal Pulsar distribution -
https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
* Updates on Presto connector for PIP-62 -
https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk

Re: WebSite build CI job does work - blocker for announcing release !

2021-12-21 Thread Nicolò Boschi

I think the problem is that it times out (currently there's a job timeout
of 2h)
I created this PR to increase it to 3h
https://github.com/apache/pulsar/pull/13449


Il giorno mer 22 dic 2021 alle ore 08:23 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

> Hello,
> the WebSite build CI job is stuck
> https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true
>
> This is probably the problem, it is due to "crowdin" that is timing out
>
> File 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
> SKIPPED
> File 'master/website/i18n/en.json'/ - - OK
> Done in 2452.38s.
> + yarn run crowdin-download
> yarn run v1.22.5
> warning package.json: No license field
> $ crowdin --config ../crowdin.yaml download -b master
> Error: The operation was canceled.
>
>
> Can anyone help ?
> we are stuck and if we do not update the website we cannot announce the
> releases with the security fixes
>
>
> Enrico
>


-- 
Nicolò Boschi

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

2021-12-21 Thread Enrico Olivelli

Lari,

Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari 
ha scritto:

> Dear Pulsar community members,
>
> PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
> repositories" was created in April 2020. The repositories for
> pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
> ago [2].
>
> I'd like to propose that we continue with the PIP-62 plan asap, before
> Apache Pulsar 2.10.0 is released.
>
> I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> waiting for the Trino Pulsar PR [3] being completed.
>
> I am volunteering to making the changes as per the PIP-62 plan.
> When can we proceed with the plan? Do we need an explicit vote on the
> mailing list about this?
>

We discussed this many times in the past year.
I believe that there is no need to wait

Enrico



>
> BR,
>
> Lari
>
> [1]
>
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> [2]
>
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> [3] https://github.com/trinodb/trino/pull/8020
>
> Other email threads:
> * [Discuss] Don't include presto/trino in the normal Pulsar distribution -
> https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> * Updates on Presto connector for PIP-62 -
> https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
>

OSX packages status....broken ?!?

PR merge

RE: [VOTE] Apache Pulsar 2.8.2 candidate 2

[VOTE] Pulsar Release 2.7.4 Candidate 2

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

Re: [DISCUSS] Release Pulsar 2.7.4

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

Re: [DISCUSSION] PIP-124: Create init subscription before sending message to DLQ

Re: [PR] Pulsar non root docker image

Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

Re: [DISCUSSION] PIP-120: Enable client memory limit by default

[VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[VOTE] PIP-120: Enable client memory limit by default

Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults

[VOTE] PIP-117: Change Pulsar standalone defaults

[VOTE] PIP-123: Introduce Pulsar metadata CLI tool

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

Re: [VOTE] PIP-120: Enable client memory limit by default

Re: [PR] Pulsar non root docker image

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

WebSite build CI job does work - blocker for announcing release !

Re: [PR] Pulsar non root docker image

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

Re: WebSite build CI job does work - blocker for announcing release !

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

31 matches

Site Navigation

Mail list logo

Footer information