Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Hello folks,
Yesterday we committed the release notes for 2.9.0.
I just have to publish a couple of other artifacts and update the website
before announcing 2.9.0.
My plan is to complete the procedure next week.

In the meantime, early next week, I believe it is time to prepare the first
RC of 2.9.1, due to the log4j bug.

If you are aware of problems on branch-2.9 or things to be cherry-picked
because they are blocker please let me know.

Otherwise if branch-2.9 is stable I will cut the RC from what we already
have now.

I am volunteering as RM for 2.9.1 as I followed 2.9.0 and release is
basically non stable due to the bugs we discovered after completing the
VOTE and publishing the artifacts to dockerhub and Maven

Best regards

Enrico

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

I am starting 2.9.1 on Monday

Enrico

Il Dom 12 Dic 2021, 02:19 陳智弘  ha scritto:

> Totally agree
>
> PengHui Li  於 2021年12月12日 週日 08:28 寫道：
>
> > +1
> >
> > Penghui
> >
> > Matteo Merli 于2021年12月11日 周六15:28写道：
> >
> > > At this point, if 2.9.0 is non stable, I think we should fast-forward
> > > to 2.9.1 which will include security fix. Though, we should start
> > > 2.9.1 right now.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Fri, Dec 10, 2021 at 11:23 PM Michael Marshall <
> mmarsh...@apache.org>
> > > wrote:
> > > >
> > > > +1 - thanks Enrico.
> > > >
> > > > - Michael
> > > >
> > > > On Sat, Dec 11, 2021 at 1:11 AM Lari Hotari 
> > wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > la 11. jouluk. 2021 klo 9.07 Enrico Olivelli 
> > > > > kirjoitti:
> > > > >
> > > > > > Hello folks,
> > > > > > Yesterday we committed the release notes for 2.9.0.
> > > > > > I just have to publish a couple of other artifacts and update the
> > > website
> > > > > > before announcing 2.9.0.
> > > > > > My plan is to complete the procedure next week.
> > > > > >
> > > > > > In the meantime, early next week, I believe it is time to prepare
> > > the first
> > > > > > RC of 2.9.1, due to the log4j bug.
> > > > > >
> > > > > > If you are aware of problems on branch-2.9 or things to be
> > > cherry-picked
> > > > > > because they are blocker please let me know.
> > > > > >
> > > > > > Otherwise if branch-2.9 is stable I will cut the RC from what we
> > > already
> > > > > > have now.
> > > > > >
> > > > > > I am volunteering as RM for 2.9.1 as I followed 2.9.0 and release
> > is
> > > > > > basically non stable due to the bugs we discovered after
> completing
> > > the
> > > > > > VOTE and publishing the artifacts to dockerhub and Maven
> > > > > >
> > > > > > Best regards
> > > > > >
> > > > > > Enrico
> > > > > >
> > >
> >
>

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Dave,
You are correct.
Pulsar 2.9.0 has already been released and also some people already started
to report issues.
The docker images have been deployed and we cannot change them.

I am finishing the release process for 2.9.0 with the website updates.

I am preparing 2.9.1.

I propose to just skip the announcement for 2.9.0.

If we are quick during the VOTE we can close this story within the end of
the week

Enrico

Il Lun 13 Dic 2021, 06:34 Dave Fisher  ha scritto:

> (1) we have published 2.9.0 at
> https://downloads.apache.org/pulsar/pulsar-2.9.0/
>
> (2) we have published 2.9.0 artifacts through maven central. They don’t
> let anyone republish versions.
>
> There are no do overs on versions. We simply cannot redo 2.9.0 at this
> moment.
>
> All the best,
> Dave
>
> Sent from my iPhone
>
> > On Dec 12, 2021, at 8:49 PM, Sijie Guo  wrote:
> >
> > My take is - if we haven't announced 2.9, I would suggest just redoing
> the
> > 2.9.0 release.
> >
> > - Sijie
> >
> >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen  wrote:
> >>
> >> I am a little confused about why we should skip 2.9.0 and not continue
> >> to release 2.9.0 by including the critical bug fixes. In fact, the
> >> Pulsar 2.9.0 release is not yet completed.
> >>
> >> For users, they will worry about whether the Pulsar release process is
> >> standardized if we skip 2.9.0. They will also worry about the release
> >> quality of Apache Pulsar if we have found the critical bugs before it
> >> is released but not included it into the release version. For Pulsar
> >> 2.9.0, it couldn't be deployed into the production environment due to
> >> the critical bug https://github.com/apache/pulsar/pull/12993
> >>
> >> Regards,
> >> Hang
> >>
> >> Dave Fisher  于2021年12月13日周一 09:40写道：
> >>>
> >>> It can be the case that releases are not announced. For example with
> >> Tomcat a version which fails to pass the vote is skipped.
> >>>
> >>> Let’s not announce 2.9.0 and go on to 2.9.1.
> >>>
> >>> Maybe there’s some website fixes to hide 2.9.0.
> >>>
> >>>
> >>> Sent from my iPhone
> >>>
> >>>> On Dec 12, 2021, at 5:28 PM, PengHui Li  wrote:
> >>>>
> >>>> Another point is we have not announced the 2.9.0 release yet.
> >>>>
> >>>> This will make users feel confused that a new release from the Pulsar
> >>>> community with the
> >>>> serious problem(log4j bug) but after the log4j has fixed the issue and
> >>>> provided the new release.
> >>>>
> >>>> I think we'd better contain the fix in 2.9.0 and 2.9.0 also has a
> >> critical
> >>>> bug https://github.com/apache/pulsar/pull/12993
> >>>> which will lead the topic stop to provide services for more than 5min.
> >> It
> >>>> looks like, hey, we have a new release here but
> >>>> it has critical security issues and known serious bugs which will
> >> seriously
> >>>> affect the core features.
> >>>>
> >>>> From the perspective of release, yes, the release vote has closed. But
> >> I
> >>>> believe that users will not care about this matter,
> >>>> they only care about the quality of the products we provided.
> >>>>
> >>>> I would like to hear your views.
> >>>>
> >>>> Regards,
> >>>> Penghui
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Dec 12, 2021 at 6:26 PM Enrico Olivelli  >
> >> wrote:
> >>>>>
> >>>>> I am starting 2.9.1 on Monday
> >>>>>
> >>>>> Enrico
> >>>>>
> >>>>> Il Dom 12 Dic 2021, 02:19 陳智弘  ha scritto:
> >>>>>
> >>>>>> Totally agree
> >>>>>>
> >>>>>>> PengHui Li  於 2021年12月12日 週日 08:28 寫道：
> >>>>>>
> >>>>>>> +1
> >>>>>>>
> >>>>>>> Penghui
> >>>>>>>
> >>>>>>> Matteo Merli 于2021年12月11日 周六15:28写道：
> >>>>>>>
> >>>>>>>> At this point, if 2.9.0 is non stable, I think we should
> >> fast-forward
> >>>>>>>> to 2.9.1 which will include security fix. Though, we should start
>

Re: PIP 112: Generate Release Notes Automatically

[Enrico Olivelli]

Yu,
thanks a great initiative and I support it at 100%

It looks like you are on your way.
Looking forward to seeing the results !

Enrico


Il giorno lun 13 dic 2021 alle ore 09:19 Yu  ha scritto:

> Hi Pulsarers,
>
> As we know[1], there are some issues in the current Pulsar release notes
> (RN), for example:
>
> - For Pulsar users
> They cannot capture the highlights quickly since the RN is a raw dump of
> PRs.
>
> - For Pulsar release managers (RM)
> They feel overwhelmed by the **manual** workload of generating RN since it
> is created based on git commit messages, while many people do not provide
> clear and meaningful info.
> It’s time-consuming to clear up all info especially for a major release
> with lots of PRs.
>
> If RN is regarded as an afterthought and finished as a last-minute task, it
> is likely not written well.
> Instead of rushing, treating RN as a part of development not only reduces
> RM's workload and makes communication more coordinated,
> but also allows more time for us to choose the most valuable highlights
> shown to users.
> Consequently, the process of the current workflow should be improved.
>
> Therefore, I propose the PIP 112: Generate Release Notes Automatically [2]
> and add some initial thoughts and research there.
> It is only a draft but I would like to invite you to join us to bring
> another major change to Pulsar. I believe this would bring many benefits to
> all of us, thanks!
>
> [1] https://lists.apache.org/thread/dl3jb9p3zvlc6ntlkpmxf1m8dw5dcd8z
> [2]
>
> https://github.com/apache/pulsar/wiki/PIP-112%3A-Generate-Release-Notes-Automatically
>

CI is failing consistently due to maven.restlet.com

[Enrico Olivelli]

Hello,
CI is failing due to an expired certificate on maven.restlet.com.

I was waiting for CI to be green before cutting 2.9.1so sad !

Error: Failed to execute goal on project pulsar-io-solr: Could not resolve
dependencies for project org.apache.pulsar:pulsar-io-solr:jar:2.9.0: Failed
to collect dependencies at org.apache.solr:solr-core:jar:8.6.3 ->
org.restlet.jee:org.restlet:jar:2.4.3: Failed to read artifact descriptor
for org.restlet.jee:org.restlet:jar:2.4.3: Could not transfer artifact
org.restlet.jee:org.restlet:pom:2.4.3 from/to maven-restlet (
https://maven.restlet.com): transfer failed for
https://maven.restlet.com/org/restlet/jee/org.restlet/2.4.3/org.restlet-2.4.3.pom:
PKIX path validation failed: java.security.cert.CertPathValidatorException:
validity check failed: NotAfter: Sat Dec 11 22:28:29 UTC 2021 -> [Help 1]

Thoughts ?

Enrico

Re: CI is failing consistently due to maven.restlet.com

[Enrico Olivelli]

I would change CI jobs by adding "
-Dmaven.wagon.http.ssl.ignore.validity.dates=true"

but that should considered some kind of security problem as we could
download from bad https servers

Enrico


Il giorno lun 13 dic 2021 alle ore 11:35 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

> Hello,
> CI is failing due to an expired certificate on maven.restlet.com.
>
> I was waiting for CI to be green before cutting 2.9.1so sad !
>
> Error: Failed to execute goal on project pulsar-io-solr: Could not resolve
> dependencies for project org.apache.pulsar:pulsar-io-solr:jar:2.9.0: Failed
> to collect dependencies at org.apache.solr:solr-core:jar:8.6.3 ->
> org.restlet.jee:org.restlet:jar:2.4.3: Failed to read artifact descriptor
> for org.restlet.jee:org.restlet:jar:2.4.3: Could not transfer artifact
> org.restlet.jee:org.restlet:pom:2.4.3 from/to maven-restlet (
> https://maven.restlet.com): transfer failed for
> https://maven.restlet.com/org/restlet/jee/org.restlet/2.4.3/org.restlet-2.4.3.pom:
> PKIX path validation failed: java.security.cert.CertPathValidatorException:
> validity check failed: NotAfter: Sat Dec 11 22:28:29 UTC 2021 -> [Help 1]
>
> Thoughts ?
>
> Enrico
>

Re: [discuss] BacklogQuota param change

[Enrico Olivelli]

ZhangJian

it looks like a regression,
I will wait for a patch before cutting the first RC for 2.9.1

thanks for sharing this problem

Enrico

Il giorno mar 14 dic 2021 alle ore 07:19 PengHui Li  ha
scritto:

> Thanks, ZhangJian.
>
> Looking forward to your PR.
>
> Penghui
>
>
> On Tue, Dec 14, 2021 at 1:40 PM ZhangJian He  wrote:
>
> > After discussing it with Matteo. It’s prob not backward compatible.
> >
> > I will work on a fix.
> >
> > Thanks
> > ZhangJian He
> >
> > ZhangJian He  于2021年12月14日周二 12:57写道：
> >
> > > I found a change between pulsar2.7 and pulsar2.9
> > >
> > > command
> > > ```
> > > bin/pulsar admin namespaces get-backlog-quotas $tenant/$namespace
> > > ```
> > >
> > > pulsar2.7 returns `limit=1024, policy=consumer_backlog_eviction`
> > >
> > > but pulsar 2.9 returns
> > >
> > > ```[root@23da8000c7c1 bin]# ./pulsar-admin namespaces
> get-backlog-quotas
> > > public/functions
> > > "destination_storageBacklogQuotaImpl(limitSize=102400,
> limitTime=-1,
> > > policy=consumer_backlog_eviction)"
> > > ```
> > >
> > > I found that the @JsonAlias("limit") annotation has been removed
> > > on org.apache.pulsar.common.policies.data.BacklogQuota in
> > > https://github.com/apache/pulsar/pull/10774.
> > > My question is, Is that a expected change or compatible change? I
> search
> > > the 2.8.0 release notes, I didn't saw it.
> > >
> > > And If it involves work, I'm happy to help :)
> > >
> > >
> > > Thanks
> > > ZhangJian He
> > >
> >
>

Re: [ANNOUNCE] New Committer: Marvin Cai

[Enrico Olivelli]

Kudos !!

Welcome aboard

Enrico

Il giorno mar 14 dic 2021 alle ore 09:12 guo jiwei 
ha scritto:

> Congratulations Marvin!
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Tue, Dec 14, 2021 at 3:44 PM Yu  wrote:
>
> > Congratulations Marvin!
> >
> > On Tue, Dec 14, 2021 at 8:42 AM Guangning E 
> wrote:
> >
> > > Configrats!
> > >
> > > Thanks,
> > > Guangning
> > >
> > > Huanli Meng  于2021年12月14日周二 08:34写道：
> > >
> > > > Congratulations Marvin!
> > > >
> > > > BR//Huanli
> > > >
> > > > > On Dec 13, 2021, at 5:46 PM, linlin  wrote:
> > > > >
> > > > > The Apache Pulsar Project Management Committee (PMC) has invited
> > Marvin
> > > > Cai
> > > > > https://github.com/MarvinCai to become a committer and we are
> > pleased
> > > to
> > > > > announce that he has accepted.
> > > > >
> > > > > Marvin has joined the community for more than 1 year now and he is
> > > > active in
> > > > > the Pulsar community for more than 6 months.
> > > > >
> > > > > Welcome and Congratulations, Marvin!
> > > > >
> > > > > Please join us in congratulating and welcoming Marvin onboard!
> > > > >
> > > > > Best Regards,
> > > > > Lin Lin on behalf of the Pulsar PMC
> > > >
> > > >
> > >
> >
>

Re: [PIP-78] Reduce redundant producers from partitioned producer

[Enrico Olivelli]

sorry for the late reply, I missed this message.


Il giorno mar 7 dic 2021 alle ore 05:56 Yuri Mizushima <
yumiz...@yahoo-corp.jp> ha scritto:

> Enrico,
> Thank you for your comment.
>
> > IIUC with this change the client will control which metrics are reported
> by
> > the broker ?
>
> From the protocol perspective, yes.
> However, the main point of this change is not to "control" metrics by the
> client side,
> but to make the broker aggregate partitioned topic's producer metrics
> explicitly.
>
> Do you suggest adding a broker config that
> configures whether partitioned producer stats are aggregated by
> producerName instead of
> introducing a backward compatibility key (i.e.,
> partial_producer_supported) on the client-side?
>

Not exactly.

I would add a configuration parameter to allow the client to use this
feature.
So:
- feature enabled -> everything works like in your patch
- feature disabled -> the broker ignores *partial_producer_supported *and
runs as before

This way if you want to let the client aggregate the stats you can do it
but by default clients are not able to interact with the metrics format.

In multi-tenancy environments, like in large clusters shared by many teams,
you do not want that the client is able to change the way the broker
reports its metrics
or in general that the client is able to alter the monitoring systems.

Enrico


>
> It is simple. However, I think we can't enable the config until all
> clients are updated.
>
> What do you think?
> Regards,
> --
> Yuri Mizushima
> yumiz...@yahoo-corp.jp
>
>
> On 2021/12/02 17:37, "Enrico Olivelli"  wrote:
>
> Yuri,
> IIUC with this change the client will control which metrics are
> reported by
> the broker ?
>
> I am not sure it is a good idea, because metrics are usually managed
> by the
> owners of the brokers, who sometimes are not the same who run the
> clients.
>
> Also, I am not sure if this way it is possible for the client to force
> the
> Broker to create many metrics and create some kind of damage.
>
> Would it be better to add a Broker configuration flag to turn on this
> feature ? I mean to allow the client to select the type of metrics ?
>
>
> Enrico
>
>
> Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima <
> yumiz...@yahoo-corp.jp> ha scritto:
>
> > Do you have any comments?
> > If there are no comments by Dec. 7, I will close the discussion and
> rebase
> > the PR commit to current master.
> >
> > Regards,
> >
> > --
> > Yuri Mizushima
> > yumiz...@yahoo-corp.jp
> >
> >
> > On 2021/11/16 15:46, "Yuri Mizushima" 
> wrote:
> >
> > Dear Pulsar community,
> >
> > I have created a new PR
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=JVOJsF5wTRoAyvy%2F8kyddBJP1XsdBv3YcIt2FP4OkHI%3D&reserved=0
> > for stats aggregation,
> > but I didn't discuss about the wire protocol change. I hope we
> will
> > discuss it here.
> >
> > Currently, partitioned producer can't aggregate by any key such
> as
> > cnx, producerId, producerName, and so on.
> > I think we need to add any aggregation system.
> > Therefore, added new aggregation policy as producerName (with
> client
> > side implementation).
> >
> > New protocol field partial_producer_supported is not used for
> stats
> > aggregation. It is used for backward compatibility.
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489&data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=L%2B4JftwNnN7dUtck8wjWhwvqKhyFpyUcOcgsoinwyfQ%3D&reserved=0
> >
> > In my understanding, if introduce new stats aggregation key to
> client
> > side,
> > need a way to determine whether the feature is enabled at clien

Re: CheckStyle Rule Change

[Enrico Olivelli]

Il Mer 15 Dic 2021, 06:38 ZhangJian He  ha scritto:

> Hello everyone, I have added checkstyle on pulsar-metadata, and
> UnusedImports check on test code. If your PR is involved in these two
> situations. Please pull the master code. Thanks
>


Got it

Thanks
Enrico

>
> Thanks
> ZhangJian He
>

Re: [DISCUSSION] PIP-120: Enable client memory limit by default

[Enrico Olivelli]

+1

Enrico

Il Mer 15 Dic 2021, 06:03 ZhangJian He  ha scritto:

> +1
>
> Thanks
> ZhangJian He
>
> Neng Lu  于2021年12月15日周三 12:52写道：
>
> > +1 (non-binding)
> >
> > On 2021/12/14 19:20:02 Matteo Merli wrote:
> > > https://github.com/apache/pulsar/issues/13306
> > >
> > >
> > > Pasted below for quoting convenience.
> > >
> > >
> > > 
> > >
> > > ## Motivation
> > >
> > > In Pulsar 2.8, we have introduced a setting to control the amount of
> > memory
> > > used by a client instance.
> > >
> > > ```java
> > > interface ClientBuilder {
> > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > }
> > > ```
> > >
> > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> is
> > being
> > > enforced.
> > >
> > > I think it's a good time for 2.10 to enable this setting by default
> and,
> > > correspondingly, to disable by default the producer queue size limit.
> > >
> > > This will simplify a lot the configuration that a producer application
> > will
> > > have to come up with, when publishing with many topic/partitions or
> > > when messages
> > > are bigger than expected.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the `ClientBuilder`, change
> > >   * `memoryLimit`: 0 -> 64 MB
> > >
> > > For the `ProducerBuilder`, changes
> > >   * `maxPendingMessages`: 1000 -> 0
> > >
> > > 64MB is picked because it's a small enough memory size that will
> > guarantee
> > > a very high producer throughput, irrespective of the individual
> messages
> > size.
> > >
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >
>

Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Enrico Olivelli]

+1

Enrico

Il Mer 15 Dic 2021, 05:27 Michael Marshall  ha
scritto:

> +1
>
> - Michael
>
> On Tue, Dec 14, 2021 at 8:59 PM Shivji Kumar Jha 
> wrote:
> >
> > +1
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Wed, 15 Dec 2021 at 06:38, Hang Chen  wrote:
> >
> > > +1
> > >
> > > Thanks,
> > > Hang
> > >
> > > PengHui Li  于2021年12月15日周三 07:51写道：
> > > >
> > > > +1
> > > >
> > > > Penghui
> > > >
> > > > On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli 
> wrote:
> > > >
> > > > > Pasted below for quoting convenience.
> > > > >
> > > > >
> > > > > 
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > The consistent hashing implementation to uniformly assign keys to
> > > consumers
> > > > > in the context of a KeyShared subscription, was introduced in
> > > > > https://github.com/apache/pulsar/pull/6791, which was released in
> > > Pulsar
> > > > > 2.6.0.
> > > > >
> > > > > While consistent hashing can use slightly more memory in certain
> > > cases, it
> > > > > is
> > > > > more suitable as a general default implementation, as it leads to a
> > > fairer
> > > > > distribution of keys across consumers, and avoiding corner cases
> that
> > > > > depend
> > > > > on the sequence of addition/removal of consumers.
> > > > >
> > > > > ## Proposed changes
> > > > >
> > > > > In 2.10 release, for the setting:
> > > > >
> > > > > ```properties
> > > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > > > splitting ranges or
> > > > > # consistent hashing to reassign keys to new consumers
> > > > > subscriptionKeySharedUseConsistentHashing=false
> > > > > ```
> > > > >
> > > > > Change its default value to `true`.
> > > > >
> > > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users
> will
> > > still
> > > > > be
> > > > > able to use the old implementation.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > 
> > > > >
> > >
>

Re: [VOTE] Apache Pulsar 2.8.2 candidate 1

[Enrico Olivelli]

Lin Lin,
-1 (binding)

I have verified the packages, all good:
- unit tests are passing (many flaky tests)
- RAT passes
- build passes (JDK11 on Ubuntu)
- Pulsar standalone works (basic smoke tests)


But I am not able to verify the signatures.
Please check the commands you used to stage the release, probably you used
the wrong key

I have imported the key from
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

This is the output...

Enrico

gpg --verify apache-pulsar-2.8.2-bin.tar.gz.asc
gpg: assuming signed data in 'apache-pulsar-2.8.2-bin.tar.gz'
gpg: Signature made Wed  1 Dec 04:55:01 2021 CET
gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9
gpg: BAD signature from "linlin " [unknown]
gpg: assuming signed data in 'apache-pulsar-2.8.2-src.tar.gz'
gpg: Signature made Wed  1 Dec 04:55:06 2021 CET
gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9
gpg: BAD signature from "linlin " [unknown]

gpg: assuming signed data in 'apache-pulsar-offloaders-2.8.2-bin.tar.gz'

gpg: Signature made Wed  1 Dec 04:55:03 2021 CET

gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9

gpg: BAD signature from "linlin " [unknown]

Il giorno mar 14 dic 2021 alle ore 11:25 Massimiliano Mirelli <
massimilianomirelli...@gmail.com> ha scritto:

> Thank you for the rc!
>
> I also found the same problem mentioned by Nicolo, when running:
>
> sha512sum -c apache-pulsar-2.8.2-bin.tar.gz
>
> I get the error:
>
> sha512sum: WARNING: 1 computed checksum did NOT match
>
> Also, when verifying the GPG keys, as well, running:
>
> gpg --verify apache-pulsar-2.8.2-bin.tar.gz
>
> throws:
>
> gpg: BAD signature from "linlin " [unknown]
>
> On Tue, 14 Dec 2021 at 10:47, Nicolò Boschi  wrote:
>
> > Thank you for driving the release!
> >
> > I found out a problem with the checksum of apache-pulsar-2.8.2-bin.tar.gz
> >
> > You wrote in the email the sha512 is f51e93d5[..]683 and actually it is
> > correct (shasum -a 512 apache-pulsar-2.8.2-bin.tar.gz
> > But this file (
> >
> >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/apache-pulsar-2.8.2-bin.tar.gz.sha512
> > )
> > reports another checksum
> >
> > Can you please check and confirm?
> >
> >
> > Il giorno mar 14 dic 2021 alle ore 04:26 linlin  ha
> > scritto:
> >
> > > This is the first release candidate for Apache Pulsar, version 2.8.2.
> > >
> > > It fixes the following issues:
> > >
> > >
> >
> https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+is%3Aclosed+label%3Arelease%2F2.8.2
> > >
> > > *** Please download, test and vote on this release. This vote will stay
> > > open
> > > for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> for
> > > convenience.
> > >
> > > Source and binary files:
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/
> > >
> > > SHA-512 checksums:
> > >
> > >
> >
> f51e93d5caa7ea4ec2616e096ca75dd71bccb475632ee5ff35d713b8f5112689d17315a1cd9350dd8f8f0bdc2e059be5fb179b2b8b3b39aae77e466103294683
> > >  apache-pulsar-2.8.2-bin.tar.gz
> > >
> > >
> >
> 8540641e76fb541f9dbfaff263946ed19a585266e5de011e78188d78ec4e1c828e8893eb2e783a1ebad866f5513efffd93396b7abd77c347f34ab689badf4fad
> > >  apache-pulsar-2.8.2-src.tar.gz
> > >
> > >
> > > Maven staging repo:
> > >
> https://repository.apache.org/content/repositories/orgapachepulsar-1108/
> > >
> > > The tag to be voted upon:
> > > v2.8.2-candidate-1
> > > https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-1
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download the source package, and follow the README to build
> > > and run the Pulsar standalone service.
> > >
> > > Lin Lin
> > >
> >
> >
> > --
> > Nicolò Boschi
> >
>

Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults

[Enrico Olivelli]

+1

@Haiting
Pulsar Standalone is usually used for local development, I am not sure you
need a migration tool.
You can still keep the configuration to use previously (standalone.conf)

Enrico

Il giorno mer 15 dic 2021 alle ore 10:56 Haiting Jiang <
jianghait...@apache.org> ha scritto:

> +1 (non-binding)
>
> Out of the scope of this PIP, a follow up data migration tool could be
> added to help migration from old settings without data loss.
>
> ---
> Haiting
>
> On 2021/12/15 01:04:21 Hang Chen wrote:
> > +1
> >
> > Thanks,
> > Hang
> >
> > PengHui Li  于2021年12月15日周三 07:52写道：
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli 
> wrote:
> > >
> > > > https://github.com/apache/pulsar/issues/13302
> > > >
> > > > Copying here for quoting convenience
> > > > 
> > > >
> > > >
> > > >
> > > >
> > > > ## Motivation
> > > >
> > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar
> cluster,
> > > > where
> > > > all the components are started within the context of a single JVM
> process.
> > > >
> > > > Users are using the standalone as a way to get quickly started with
> Pulsar
> > > > or
> > > > in all the cases where it makes sense to have a single node
> deployment.
> > > >
> > > > Right now, the standalone is starting by default with many
> components,
> > > > several of
> > > > which are quite complex, since they are designed to be deployed in a
> > > > distributed
> > > > fashion.
> > > >
> > > > ## Goal
> > > >
> > > > Simplify the components of Pulsar standalone to achieve:
> > > >
> > > >  1. Reduce complexity
> > > >  2. Reduce startup time
> > > >  3. Reduce memory and CPU footprint of running standalone
> > > >
> > > > ## Proposed changes
> > > >
> > > > The proposal here is to change some of the default implementations
> that are
> > > > used for the Pulsar standalone.
> > > >
> > > >  1. **Metadata Store implementation** -->
> > > >   Change from ZooKeeper to RocksDB
> > > >
> > > >  2. **Pulsar functions package backend** -->
> > > >   Change from using DistributedLog to using local filesystem,
> storing
> > > > the
> > > >   jars directly in the data folder instead of uploading them
> into BK.
> > > >
> > > >  3. **Pulsar functions state store implementation** -->
> > > >   Change the state store to be backed by a MetadataStore based
> backed,
> > > >   with the RocksDB implementation.
> > > >
> > > >  4. **Table Service** -->
> > > >   Do not start BK table service by default
> > > >
> > > > ## Compatibility considerations
> > > >
> > > > In order to avoid compatibility issues where users have existing
> Pulsar
> > > > standalone services that they want to upgrade without conflicts, we
> will
> > > > follow the principle of keeping the old defaults where there is
> existing
> > > > data on the disk.
> > > >
> > > > We will add a file, serving the purpose as a flag, in the
> `data/standalone`
> > > > directory, for example `new-2.10-defaults`.
> > > >
> > > > If the file is present, or if the data directory is completely
> missing, we
> > > > will adopt the new set of default configuration settings.
> > > >
> > > > If the file is not there, we will continue to use existing defaults
> and we
> > > > will
> > > > not break the upgrade operation.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
> >
>

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

;
> > >>>>>> Maybe there’s some website fixes to hide 2.9.0.
> > >>>>>>
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>>
> > >>>>>>> On Dec 12, 2021, at 5:28 PM, PengHui Li 
> > wrote:
> > >>>>>>>
> > >>>>>>> Another point is we have not announced the 2.9.0 release yet.
> > >>>>>>>
> > >>>>>>> This will make users feel confused that a new release from the
> > >> Pulsar
> > >>>>>>> community with the
> > >>>>>>> serious problem(log4j bug) but after the log4j has fixed the
> issue
> > >> and
> > >>>>>>> provided the new release.
> > >>>>>>>
> > >>>>>>> I think we'd better contain the fix in 2.9.0 and 2.9.0 also has a
> > >>>>> critical
> > >>>>>>> bug https://github.com/apache/pulsar/pull/12993
> > >>>>>>> which will lead the topic stop to provide services for more than
> > >> 5min.
> > >>>>> It
> > >>>>>>> looks like, hey, we have a new release here but
> > >>>>>>> it has critical security issues and known serious bugs which will
> > >>>>> seriously
> > >>>>>>> affect the core features.
> > >>>>>>>
> > >>>>>>> From the perspective of release, yes, the release vote has
> closed.
> > >> But
> > >>>>> I
> > >>>>>>> believe that users will not care about this matter,
> > >>>>>>> they only care about the quality of the products we provided.
> > >>>>>>>
> > >>>>>>> I would like to hear your views.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Penghui
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Sun, Dec 12, 2021 at 6:26 PM Enrico Olivelli <
> > >> eolive...@gmail.com
> > >>>>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> I am starting 2.9.1 on Monday
> > >>>>>>>>
> > >>>>>>>> Enrico
> > >>>>>>>>
> > >>>>>>>> Il Dom 12 Dic 2021, 02:19 陳智弘  ha
> scritto:
> > >>>>>>>>
> > >>>>>>>>> Totally agree
> > >>>>>>>>>
> > >>>>>>>>>> PengHui Li  於 2021年12月12日 週日 08:28 寫道：
> > >>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>> Penghui
> > >>>>>>>>>>
> > >>>>>>>>>> Matteo Merli 于2021年12月11日 周六15:28写道：
> > >>>>>>>>>>
> > >>>>>>>>>>> At this point, if 2.9.0 is non stable, I think we should
> > >>>>> fast-forward
> > >>>>>>>>>>> to 2.9.1 which will include security fix. Though, we should
> > >> start
> > >>>>>>>>>>> 2.9.1 right now.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Matteo Merli
> > >>>>>>>>>>> 
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Dec 10, 2021 at 11:23 PM Michael Marshall <
> > >>>>>>>>> mmarsh...@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +1 - thanks Enrico.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Michael
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sat, Dec 11, 2021 at 1:11 AM Lari Hotari <
> > >> lhot...@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> +1
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> la 11. jouluk. 2021 klo 9.07 Enrico Olivelli <
> > >>>>>>>> eolive...@gmail.com>
> > >>>>>>>>>>>>> kirjoitti:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello folks,
> > >>>>>>>>>>>>>> Yesterday we committed the release notes for 2.9.0.
> > >>>>>>>>>>>>>> I just have to publish a couple of other artifacts and
> > update
> > >>>>>>>> the
> > >>>>>>>>>>> website
> > >>>>>>>>>>>>>> before announcing 2.9.0.
> > >>>>>>>>>>>>>> My plan is to complete the procedure next week.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> In the meantime, early next week, I believe it is time to
> > >>>>>>>> prepare
> > >>>>>>>>>>> the first
> > >>>>>>>>>>>>>> RC of 2.9.1, due to the log4j bug.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If you are aware of problems on branch-2.9 or things to be
> > >>>>>>>>>>> cherry-picked
> > >>>>>>>>>>>>>> because they are blocker please let me know.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Otherwise if branch-2.9 is stable I will cut the RC from
> > what
> > >>>>>>>> we
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>>> have now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I am volunteering as RM for 2.9.1 as I followed 2.9.0 and
> > >>>>>>>> release
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>> basically non stable due to the bugs we discovered after
> > >>>>>>>>> completing
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> VOTE and publishing the artifacts to dockerhub and Maven
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best regards
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Enrico
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >>
> > >> Chris Herzog Messaging Team | O 630 300 7718 | M 815 263 3764 |
> > >> www.tibco.com
> > >>
> > >> <http://www.tibco.com/>
> > >>
> >
> >
>

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Il giorno mer 15 dic 2021 alle ore 12:52 Hang Chen  ha
scritto:

> Hi Enrico,
>  Thanks for your great work!  I found 150+ PR labeled as
> `release/2.9.1`, but doesn't contain in v2.9.1-candidate-1 tag. Does
> those PRs release in next version?
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.1+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed
>
>
In theory the answer is "yes",
I am double checking.
it is always an hard task

Enrico


> Thanks,
> Hang
>
> Enrico Olivelli  于2021年12月15日周三 19:19写道：
> >
> > I had prepared the rc1 for 2.9.1
> >
> > but today I added https://github.com/apache/pulsar/pull/13291
> >
> > I will create a new RC and send a VOTE, possibly today
> >
> > The first RC will be rc2, in order to not mess up the git repository and
> > the dist area
> >
> > Enrico
> >
> > Il giorno lun 13 dic 2021 alle ore 19:22 Sijie Guo 
> ha
> > scritto:
> >
> > > Thank you for sharing that!
> > >
> > > I think we should separate discussing a process from finishing a
> release.
> > > In other words, we shouldn't block on a process in order to finish a
> > > release.
> > >
> > > We should use the old process to finish a release while discussing a
> > > process to improve the release notes process.
> > >
> > > - Sijie
> > >
> > > On Mon, Dec 13, 2021 at 10:04 AM Dave Fisher  wrote:
> > >
> > > >
> > > >
> > > > > On Dec 13, 2021, at 9:57 AM, Sijie Guo  wrote:
> > > > >
> > > > > I am fine with doing 2.9.1.
> > > > >
> > > > > I am trying to understand what happened between released 2.9.0 and
> > > > > announcing it.
> > > > >
> > > > > It usually means there is a gap in the release process. We need to
> > > solve
> > > > > the process. If it is RM's responsibility for announcing the
> release,
> > > it
> > > > > should happen as soon as the release was cut. If the RM doesn't do
> it
> > > in
> > > > > time, other committers or PMC members should jump on it to help. I
> feel
> > > > > something was held up somewhere. But I don't know what is going on
> > > there.
> > > >
> > > > See the thread regarding release notes -
> > > > https://lists.apache.org/thread/sszycc3zjxkdqd9x5f16108qn0x7w5g1
> > > >
> > > > Regards,
> > > > Dave
> > > > >
> > > > > - Sijie
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Dec 13, 2021 at 9:47 AM Chris Herzog
>  > > >
> > > > > wrote:
> > > > >
> > > > >> I'm 100% with Dave.  2.9.0 is released (it's up on Maven), if
> it's not
> > > > >> "announced", that's just a "publicity" effort because the 2.9.0
> > > release
> > > > is
> > > > >> out there.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sun, Dec 12, 2021 at 11:34 PM Dave Fisher <
> wave4d...@comcast.net>
> > > > >> wrote:
> > > > >>
> > > > >>> (1) we have published 2.9.0 at
> > > > >>> https://downloads.apache.org/pulsar/pulsar-2.9.0/
> > > > >>>
> > > > >>> (2) we have published 2.9.0 artifacts through maven central. They
> > > don’t
> > > > >>> let anyone republish versions.
> > > > >>>
> > > > >>> There are no do overs on versions. We simply cannot redo 2.9.0 at
> > > this
> > > > >>> moment.
> > > > >>>
> > > > >>> All the best,
> > > > >>> Dave
> > > > >>>
> > > > >>> Sent from my iPhone
> > > > >>>
> > > > >>>> On Dec 12, 2021, at 8:49 PM, Sijie Guo 
> wrote:
> > > > >>>>
> > > > >>>> My take is - if we haven't announced 2.9, I would suggest just
> > > > redoing
> > > > >>> the
> > > > >>>> 2.9.0 release.
> > > > >>>>
> > > > >>>> - Sijie
> > > > >>>>
> > > > >>>>> On Sun, Dec 12, 2021 at 6:35 PM

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Hang,
sorry, I wanted to say that I am double checking if people forgot to add
"cherry-picked"

Enrico

Il giorno mer 15 dic 2021 alle ore 13:20 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

>
>
> Il giorno mer 15 dic 2021 alle ore 12:52 Hang Chen 
> ha scritto:
>
>> Hi Enrico,
>>  Thanks for your great work!  I found 150+ PR labeled as
>> `release/2.9.1`, but doesn't contain in v2.9.1-candidate-1 tag. Does
>> those PRs release in next version?
>>
>> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.1+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed
>>
>>
> In theory the answer is "yes",
> I am double checking.
> it is always an hard task
>
> Enrico
>
>
>> Thanks,
>> Hang
>>
>> Enrico Olivelli  于2021年12月15日周三 19:19写道：
>> >
>> > I had prepared the rc1 for 2.9.1
>> >
>> > but today I added https://github.com/apache/pulsar/pull/13291
>> >
>> > I will create a new RC and send a VOTE, possibly today
>> >
>> > The first RC will be rc2, in order to not mess up the git repository and
>> > the dist area
>> >
>> > Enrico
>> >
>> > Il giorno lun 13 dic 2021 alle ore 19:22 Sijie Guo 
>> ha
>> > scritto:
>> >
>> > > Thank you for sharing that!
>> > >
>> > > I think we should separate discussing a process from finishing a
>> release.
>> > > In other words, we shouldn't block on a process in order to finish a
>> > > release.
>> > >
>> > > We should use the old process to finish a release while discussing a
>> > > process to improve the release notes process.
>> > >
>> > > - Sijie
>> > >
>> > > On Mon, Dec 13, 2021 at 10:04 AM Dave Fisher  wrote:
>> > >
>> > > >
>> > > >
>> > > > > On Dec 13, 2021, at 9:57 AM, Sijie Guo 
>> wrote:
>> > > > >
>> > > > > I am fine with doing 2.9.1.
>> > > > >
>> > > > > I am trying to understand what happened between released 2.9.0 and
>> > > > > announcing it.
>> > > > >
>> > > > > It usually means there is a gap in the release process. We need to
>> > > solve
>> > > > > the process. If it is RM's responsibility for announcing the
>> release,
>> > > it
>> > > > > should happen as soon as the release was cut. If the RM doesn't
>> do it
>> > > in
>> > > > > time, other committers or PMC members should jump on it to help.
>> I feel
>> > > > > something was held up somewhere. But I don't know what is going on
>> > > there.
>> > > >
>> > > > See the thread regarding release notes -
>> > > > https://lists.apache.org/thread/sszycc3zjxkdqd9x5f16108qn0x7w5g1
>> > > >
>> > > > Regards,
>> > > > Dave
>> > > > >
>> > > > > - Sijie
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Dec 13, 2021 at 9:47 AM Chris Herzog
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > >> I'm 100% with Dave.  2.9.0 is released (it's up on Maven), if
>> it's not
>> > > > >> "announced", that's just a "publicity" effort because the 2.9.0
>> > > release
>> > > > is
>> > > > >> out there.
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Sun, Dec 12, 2021 at 11:34 PM Dave Fisher <
>> wave4d...@comcast.net>
>> > > > >> wrote:
>> > > > >>
>> > > > >>> (1) we have published 2.9.0 at
>> > > > >>> https://downloads.apache.org/pulsar/pulsar-2.9.0/
>> > > > >>>
>> > > > >>> (2) we have published 2.9.0 artifacts through maven central.
>> They
>> > > don’t
>> > > > >>> let anyone republish versions.
>> > > > >>>
>> > > > >>> There are no do overs on versions. We simply cannot redo 2.9.0
>> at
>> > > this
>> > > > >>> moment.
>> > > > >>>
>> > > > >>

Re: Does the #13291 should contains in 2.8.2

[Enrico Olivelli]

+1
Enrico

Il giorno mer 15 dic 2021 alle ore 14:27 PengHui Li  ha
scritto:

> We should contain #13291 in 2.8.2.
>
> Penghui
>
> On Wed, Dec 15, 2021 at 7:19 PM ZhangJian He  wrote:
>
> > Hello,
> > @LinLin @lipenghui
> >
> > I want to discuss if #13291 should contains in release 2.8.2.
> >
> > It's a compatible problem since 2.8, and we already commited it to master
> > and branch 2.9
> >
> >
> > Thanks
> > ZhangJian He
> >
>

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Devin,

Il giorno mer 15 dic 2021 alle ore 15:32 Devin Bost 
ha scritto:

> If there are known critical bugs in 2.9.0, we should be transparent with
> users (and let them know that it's beta, or at least mention what features
> are incomplete) if it's going to be announced so trust is not damaged.
>

Actually we found a regression in 2.9.0 right after closing the VOTE.
It also includes the upgrade of log4j2

When we closed the VOTE there was no known reason to say that the release
was not stable or that it contained vulnerabilities


Enrico




> Although I typically expect x.x.0 releases to be experimental (based on my
> experience), users without that experience might not realize that it's not
> yet safe to upgrade production environments to this version.
>
> I'd hope that all users would perform the rigorous testing required to
> discover any bugs that might appear in their production environments, but
> if some users trust the Apache releases to be stable, that trust can be
> damaged if we're not careful.
>
>
> Devin Bost
> (Sent from mobile)
>
> 
> From: Enrico Olivelli 
> Sent: Wednesday, December 15, 2021 5:29:53 AM
> To: Dev 
> Subject: Re: Status of Pulsar 2.9.0 and starting 2.9.1
>
> Hang,
> sorry, I wanted to say that I am double checking if people forgot to add
> "cherry-picked"
>
> Enrico
>
> Il giorno mer 15 dic 2021 alle ore 13:20 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> >
> >
> > Il giorno mer 15 dic 2021 alle ore 12:52 Hang Chen 
> > ha scritto:
> >
> >> Hi Enrico,
> >>  Thanks for your great work!  I found 150+ PR labeled as
> >> `release/2.9.1`, but doesn't contain in v2.9.1-candidate-1 tag. Does
> >> those PRs release in next version?
> >>
> >>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.1+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed
> >>
> >>
> > In theory the answer is "yes",
> > I am double checking.
> > it is always an hard task
> >
> > Enrico
> >
> >
> >> Thanks,
> >> Hang
> >>
> >> Enrico Olivelli  于2021年12月15日周三 19:19写道：
> >> >
> >> > I had prepared the rc1 for 2.9.1
> >> >
> >> > but today I added https://github.com/apache/pulsar/pull/13291
> >> >
> >> > I will create a new RC and send a VOTE, possibly today
> >> >
> >> > The first RC will be rc2, in order to not mess up the git repository
> and
> >> > the dist area
> >> >
> >> > Enrico
> >> >
> >> > Il giorno lun 13 dic 2021 alle ore 19:22 Sijie Guo <
> guosi...@gmail.com>
> >> ha
> >> > scritto:
> >> >
> >> > > Thank you for sharing that!
> >> > >
> >> > > I think we should separate discussing a process from finishing a
> >> release.
> >> > > In other words, we shouldn't block on a process in order to finish a
> >> > > release.
> >> > >
> >> > > We should use the old process to finish a release while discussing a
> >> > > process to improve the release notes process.
> >> > >
> >> > > - Sijie
> >> > >
> >> > > On Mon, Dec 13, 2021 at 10:04 AM Dave Fisher 
> wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > > On Dec 13, 2021, at 9:57 AM, Sijie Guo 
> >> wrote:
> >> > > > >
> >> > > > > I am fine with doing 2.9.1.
> >> > > > >
> >> > > > > I am trying to understand what happened between released 2.9.0
> and
> >> > > > > announcing it.
> >> > > > >
> >> > > > > It usually means there is a gap in the release process. We need
> to
> >> > > solve
> >> > > > > the process. If it is RM's responsibility for announcing the
> >> release,
> >> > > it
> >> > > > > should happen as soon as the release was cut. If the RM doesn't
> >> do it
> >> > > in
> >> > > > > time, other committers or PMC members should jump on it to help.
> >> I feel
> >> > > > > something was held up somewhere. But I don't know what is going
> on
> >> > > there.
> >> > > >
> >> > > > See the thread 

Re: [DISCUSS] Add definition to our cherry picking process

[Enrico Olivelli]

Jonathan,

Il Mer 15 Dic 2021, 21:11 Jonathan Ellis  ha scritto:

> What are the benefits of cherry picking vs committing to the oldest
> relevant branch and merging forwards?


We (and most of the ASF projects I am involved in) ask contributors to
target the master branch for the patches.

How can we do what you suggest?
Asking people to send PRs against the oldest branch relevant to the patch?

Probably the contributor is not aware of the other branches.

Can you please explain more your idea?

Enrico

Git is designed around merge since
> that preserves the commit sha which allows it to be tracked across
> branches.  And a merge based workflow avoids the problem here by design.
>
> On Wed, Dec 15, 2021 at 1:54 PM Michael Marshall 
> wrote:
>
> > Hello Pulsar Community,
> >
> > As far as I can tell, we do not have a publicly defined process for
> > cherry-picking commits to release branches [0]. As a new committer,
> > I'd like to provide my newly gained perspective and ask that we add
> > some guidance to our wiki.
> >
> > Regarding cherry picking, I see two patterns in Pulsar:
> > 1. Some committers cherry pick commits when they merge a PR.
> > 2. Some committers leave PRs to be cherry picked right before tagging
> > the dot release. The release manager is expected to cherry pick these
> > commits.
> >
> > Anecdotally, pattern 2 is more common.
> >
> > I am concerned that pattern 2 creates a lot of extra work for the
> > release manager, leaves the community with little time to build and
> > test release branches before tagging release candidates, and leaves us
> > unable to quickly respond to security issues.
> >
> > If we switch to pattern 1, merging PRs may become more time consuming.
> > However, it lightens the load on release managers and makes the
> > project more nimble. Further, resolving conflicts should be easier, as
> > the committer already has the PR's context, which could reduce the
> > risk of regressions.
> >
> > For example, the 2.7.4 release is delayed because of pattern 2.
> > Penghui and I cherry picked many commits over the recent days because
> > branch-2.7 was not up to date. Now, the tests aren't passing for the
> > branch, and since we shouldn't cut a release until the tests pass, we
> > haven't released the patch for the Log4Shell CVE.
> >
> > Another solution is to remove the expectation that the release manager
> > cherry picks commits. The downside here is that we will possibly miss
> > important bug fixes that would improve dot release stability.
> >
> > I propose that we try to use pattern 1 (or something close to it) for
> > our process.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/wiki/Committer-Guide
> >
>
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

[VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

This is the second release candidate for Apache Pulsar, version 2.9.1.

The first release candidate was aborted without starting a VOTE because we
had to pick up high priority dependency upgrades.

It fixes the following issues:
https://github.com/apache/pulsar/pulls?q=is%3Apr++label%3Arelease%2F2.9.1+

*** Please download, test and vote on this release. This vote will stay open
for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.1-candidate-2/

SHA-512 checksums:

5ca7d2c6a8ac51413214796481095bbde50b5bda95d8b8f2467989931b29c75e679aabcfebd82e9e3e90dd1644c580214e0a05eca8652a500f042c84cb21becd
 apache-pulsar-2.9.1-bin.tar.gz
34a1e22fb0ff2e69e7e880a9432526990610113cf89d93c953dff82cc443510dcf724eaa0e1fade82464f9bf5443655bd23bcf2064e312c4a9da70bb4c9937ba
 apache-pulsar-2.9.1-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1110

The tag to be voted upon:
v2.9.1-candidate-2 (f52ac045f41acbb6c31da21a3463df3cfbe8f1b4)
https://github.com/apache/pulsar/releases/tag/v2.9.1-candidate-2

Link to the release notes:
https://github.com/apache/pulsar/pull/13357

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build
and run the Pulsar standalone service.


Enrico Olivelli

Re: [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

I have pushed the docker images to my personal dockehub account

eolivelli/pulsar:2.9.1rc2
eolivelli/pulsar-all:2.9.1rc2

Enrico

Il Gio 16 Dic 2021, 15:57 Nicolò Boschi  ha scritto:

> +1 (non binding)
>
> Checks:
> - Checksum and signatures
> - Apache Rat check passes
> - OWASP check passes (I created this PR for fix a false positive
> https://github.com/apache/pulsar/pull/13364)
> - Compile from source w JDK11
> - Build docker image from source
> - Run Pulsar standalone and produce-consume from CLI
> - verified the presence of Log4j 2.16.0 jar in docker and tarball
>
> Il giorno gio 16 dic 2021 alle ore 14:25 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> > This is the second release candidate for Apache Pulsar, version 2.9.1.
> >
> > The first release candidate was aborted without starting a VOTE because
> we
> > had to pick up high priority dependency upgrades.
> >
> > It fixes the following issues:
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr++label%3Arelease%2F2.9.1+
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.1-candidate-2/
> >
> > SHA-512 checksums:
> >
> >
> >
> 5ca7d2c6a8ac51413214796481095bbde50b5bda95d8b8f2467989931b29c75e679aabcfebd82e9e3e90dd1644c580214e0a05eca8652a500f042c84cb21becd
> >  apache-pulsar-2.9.1-bin.tar.gz
> >
> >
> 34a1e22fb0ff2e69e7e880a9432526990610113cf89d93c953dff82cc443510dcf724eaa0e1fade82464f9bf5443655bd23bcf2064e312c4a9da70bb4c9937ba
> >  apache-pulsar-2.9.1-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1110
> >
> > The tag to be voted upon:
> > v2.9.1-candidate-2 (f52ac045f41acbb6c31da21a3463df3cfbe8f1b4)
> > https://github.com/apache/pulsar/releases/tag/v2.9.1-candidate-2
> >
> > Link to the release notes:
> > https://github.com/apache/pulsar/pull/13357
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> > Enrico Olivelli
> >
>
>
> --
> Nicolò Boschi
>

Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires

[Enrico Olivelli]

+1

Enrico

Il Ven 17 Dic 2021, 05:36 Sijie Guo  ha scritto:

> +1
>
> On Tue, Dec 14, 2021 at 10:03 AM Matteo Merli  wrote:
>
> > https://github.com/apache/pulsar/issues/13304
> >
> >
> > Pasted below for quoting convenience.
> >
> > ---
> >
> >
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and
> 2.9
> > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> > operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled stabilization in 2.9 the configuration setting that
> > allows
> > brokers to continue operating in a safe mode when the session with
> > ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data
> plane
> > will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> > any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that
> we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> > thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when a broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Change its default value to `reconnect`.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

Peng Hui,

Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li  ha
scritto:

> Checked:
>
> - Build from the src
> - Check signatures
> - Follow the validation process
>
> But when I try to verify PulsarSQL, got following exceptions:
>
> ```
> 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3
> io.prestosql.execution.StageStateMachine Stage
> 20211217_065818_1_cahiv.1 failed
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.nio.BufferUnderflowException
>  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
>  at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>  at
>
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
>  at
>
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
>  at
>
> org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485)
>  at
>
> io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
>  at
>
> io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
>  at io.prestosql.operator.Driver.processInternal(Driver.java:379)
>  at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
>  at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
>  at io.prestosql.operator.Driver.processFor(Driver.java:276)
>  at
>
> io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
>  at
>
> io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
>  at
>
> io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
>  at
> io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown
> Source)
>  at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.BufferUnderflowException
>  at java.nio.Buffer.nextGetIndex(Buffer.java:532)
>  at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417)
>  at
>
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106)
>  at
>
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49)
>  at
>
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61)
>  at
>
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58)
>  at
>
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>  at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>  at
>
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>  ... 18 more
> ```
>
> An issue can be found here https://github.com/apache/pulsar/issues/12284,
>

This doesn't look like a regression introduced in 2.9.1
As this is a security related release I would continue the release process.

What do you think ?

Enrico


> my test steps are very simple:
>
> 1. Start presto worker, `bin/pulsar sql-worker run`
> 2. Produce some messages, `bin/pulsar-client produce -m "hello" -n 10
> test_wordcount_src`
> 3. Query the data from the topic, `select * from
> pulsar."public/default"."test_wordcount_src";`
>
> Not able to query the produced data and get errors in the Pulsar SQL
> worker.
>
> Penghui
>
> On Fri, Dec 17, 2021 at 5:33 AM Matteo Merli  wrote:
>
> > +1
> >
> > Checked:
> >  * Signatures
> >  * Bin distribution:
> >  - NOTICE, README, LICENSE
> >      - Start standalone service and producer/consumer test
> >  * Src distribution:
> >  - NOTICE,  README, LICENSE
> >  - Compile and unit tests
> >  - Start standalone service
> >  * Checked staging maven repository artifacts
> >  * Checked docker images
> >
> >
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
> >
> >
> > On Thu, Dec 16, 2021 at 12:53 PM Enrico Olivelli 
> > wrote:
> > >
> > > I have pushed the docker images to my personal dockehub account
> > >
> > > eolivelli/pulsar:2.9.1rc2
> > > eolivelli/pulsar-all:2.9.1rc2
> > >
> > > Enrico
> > >
>

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Il giorno ven 17 dic 2021 alle ore 09:45 Yu  ha scritto:

> Hi Enrico,
>
> Thanks for your great effort on the 2.9.0 release.
>
> Circling back to see if there is any progress of the 2.9.0 website updates
> [1].
>
> Currently, the 2.9.0 doc is not available on the website and the 2.9.0 doc
> set has not been generated yet, any updates? Thanks
>


I will do it together with 2.9.1
because 2.9.0 is broken, so it is better to not do much buzz around it

Enrico


>
> [1]
> https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
>
>
> On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli 
> wrote:
>
> > Dave,
> > You are correct.
> > Pulsar 2.9.0 has already been released and also some people already
> started
> > to report issues.
> > The docker images have been deployed and we cannot change them.
> >
> > I am finishing the release process for 2.9.0 with the website updates.
> >
> > I am preparing 2.9.1.
> >
> > I propose to just skip the announcement for 2.9.0.
> >
> > If we are quick during the VOTE we can close this story within the end of
> > the week
> >
> > Enrico
> >
> > Il Lun 13 Dic 2021, 06:34 Dave Fisher  ha
> scritto:
> >
> > > (1) we have published 2.9.0 at
> > > https://downloads.apache.org/pulsar/pulsar-2.9.0/
> > >
> > > (2) we have published 2.9.0 artifacts through maven central. They don’t
> > > let anyone republish versions.
> > >
> > > There are no do overs on versions. We simply cannot redo 2.9.0 at this
> > > moment.
> > >
> > > All the best,
> > > Dave
> > >
> > > Sent from my iPhone
> > >
> > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo  wrote:
> > > >
> > > > My take is - if we haven't announced 2.9, I would suggest just
> redoing
> > > the
> > > > 2.9.0 release.
> > > >
> > > > - Sijie
> > > >
> > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen 
> > wrote:
> > > >>
> > > >> I am a little confused about why we should skip 2.9.0 and not
> continue
> > > >> to release 2.9.0 by including the critical bug fixes. In fact, the
> > > >> Pulsar 2.9.0 release is not yet completed.
> > > >>
> > > >> For users, they will worry about whether the Pulsar release process
> is
> > > >> standardized if we skip 2.9.0. They will also worry about the
> release
> > > >> quality of Apache Pulsar if we have found the critical bugs before
> it
> > > >> is released but not included it into the release version. For Pulsar
> > > >> 2.9.0, it couldn't be deployed into the production environment due
> to
> > > >> the critical bug https://github.com/apache/pulsar/pull/12993
> > > >>
> > > >> Regards,
> > > >> Hang
> > > >>
> > > >> Dave Fisher  于2021年12月13日周一 09:40写道：
> > > >>>
> > > >>> It can be the case that releases are not announced. For example
> with
> > > >> Tomcat a version which fails to pass the vote is skipped.
> > > >>>
> > > >>> Let’s not announce 2.9.0 and go on to 2.9.1.
> > > >>>
> > > >>> Maybe there’s some website fixes to hide 2.9.0.
> > > >>>
> > > >>>
> > > >>> Sent from my iPhone
> > > >>>
> > > >>>> On Dec 12, 2021, at 5:28 PM, PengHui Li 
> wrote:
> > > >>>>
> > > >>>> Another point is we have not announced the 2.9.0 release yet.
> > > >>>>
> > > >>>> This will make users feel confused that a new release from the
> > Pulsar
> > > >>>> community with the
> > > >>>> serious problem(log4j bug) but after the log4j has fixed the issue
> > and
> > > >>>> provided the new release.
> > > >>>>
> > > >>>> I think we'd better contain the fix in 2.9.0 and 2.9.0 also has a
> > > >> critical
> > > >>>> bug https://github.com/apache/pulsar/pull/12993
> > > >>>> which will lead the topic stop to provide services for more than
> > 5min.
> > > >> It
> > > >>>> looks like, hey, we have a new release here but
> > > >>>> it has critical security issues and known serious bugs which will
> >

Re: [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

+1 (binding)

- Run release validation procedure
- CI is passing on those sources

Enrico

Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li  ha
scritto:

> >  Will this issue be fixed in the future releases?
>
> Yes, 2.8.2 and 2.9.2 will fix the problem.
>
> Penghui
>
> On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> massimilianomirelli...@gmail.com> wrote:
>
> > Thank you for the rc!
> >
> > +1 (non-binding)
> >
> > * verify sha512 checksums
> > * verify gpg signatures
> > * build pulsar-all docker image
> > * execute Fallout distributed system test (produce / receive 10k
> messages)
> >
> > Building the docker image as indicated in the README:
> >
> > mvn clean install -DskipTests
> > mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
> >
> > I still get the error described in this PR#11951 (
> > https://github.com/apache/pulsar/pull/11951) which I suppose has been
> > cherry picked in 2.9.1.
> >
> > Giving enough permissions to /docker/pulsar/scripts/ in the src package
> and
> > then building the docker image again solved the issue.
> >
> > I also tested the problem with the image Enrico provided, this way:
> >
> > docker run -it --entrypoint bash eolivelli/pulsar-all:2.9.1rc2
> > ls -al /pulsar/bin
> >
> > and that one does have the right permissions.
> >
> > Enrico, did you build the image using the same mvn commands (^^^), or is
> > there some other way to build it?
> >
> > Thank you,
> > Max
> >
> > On Fri, 17 Dec 2021 at 16:18, 陳智弘  wrote:
> >
> > > Hi PengHu,
> > >
> > >  Will this issue be fixed in the future releases?
> > >
> > > PengHui Li  於 2021年12月17日 週五 21:53 寫道：
> > >
> > > > Hi Enrico,
> > > >
> > > > I'm ok, it only happens when the message is without a schema version.
> > > > So I'm not giving -1.
> > > >
> > > > Thanks,
> > > > Penghui
> > > >
> > > >
> > > > On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli  >
> > > > wrote:
> > > >
> > > > > Peng Hui,
> > > > >
> > > > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li <
> > > peng...@apache.org>
> > > > > ha
> > > > > scritto:
> > > > >
> > > > > > Checked:
> > > > > >
> > > > > > - Build from the src
> > > > > > - Check signatures
> > > > > > - Follow the validation process
> > > > > >
> > > > > > But when I try to verify PulsarSQL, got following exceptions:
> > > > > >
> > > > > > ```
> > > > > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3
> > > > > > io.prestosql.execution.StageStateMachine Stage
> > > > > > 20211217_065818_1_cahiv.1 failed
> > > > > > com.google.common.util.concurrent.UncheckedExecutionException:
> > > > > > java.nio.BufferUnderflowException
> > > > > >  at
> > > > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
> > > > > >  at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> > > > > >  at
> > > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> > > > > >  at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
> > > > > >  at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
> > > > > >  at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485)
> > > > > >  at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
> > > > > >  at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >

Re: [VOTE] Pulsar Release 2.7.4 Candidate 1

[Enrico Olivelli]

Jiwei Guo,
Thanks for driving the release
today is Saturday, so I am not sure how many people will have time to test
the release candidate during the 72 hours period (for instance I can do it
only on Monday, hopefully).
Please take this into consideration when you are going to close the VOTE,
maybe waiting until Tuesday may be a good idea.

Enrico



Il giorno sab 18 dic 2021 alle ore 09:54 guo jiwei 
ha scritto:

> This is the first release candidate for Apache Pulsar, version 2.7.4.
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
>
> Release note:
> https://github.com/apache/pulsar/pull/13391
>
> *** Please download, test and vote on this release. This vote will
> stay openfor at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-1/
>
> SHA-512 checksums:
>
>
> 1a34a408085328500c8c4c72e5ca9b458058a088674b1bfd550fe01ad365182b8820516f99b5c010a03037e1886ce5539cb57f5ef4a2297f1b7ed8fb5844bcd0
>  apache-pulsar-2.7.4-bin.tar.gz
>
> 37e75698946f3390a254fa2bd7d974fca27d1aed632e86d64a08dc0bf8f2e0ca830cbeed2820782e59ac6bac7368ae69b54f1f5cd3d6c2cad292166e3538e285
>  apache-pulsar-2.7.4-src.tar.gz
>
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1113
>
> The tag to be voted upon:
> v2.7.4-candidate-1 (ab451b855d873a9bad2005f939a23118a583baa9)
> https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-1
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
>
>
> Regards
> Jiwei Guo (Tboy)
>

[RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

Hello everyone,
The VOTE passed with 5 VOTEs, 3 of them were binding.

- Matteo Merli (binding)
- Enrico Olivelli (binding)
- Peng Hui (binding)
- Nicolò Boschi
- Massimiliano Mirelli

I will move forward with the next steps
https://github.com/apache/pulsar/wiki/Release-process

Enrico


Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li  ha
scritto:

> +1 binding
>
> Penghui
>
> Enrico Olivelli 于2021年12月18日 周六18:39写道：
>
> > +1 (binding)
> >
> > - Run release validation procedure
> > - CI is passing on those sources
> >
> > Enrico
> >
> > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li 
> > ha
> > scritto:
> >
> > > >  Will this issue be fixed in the future releases?
> > >
> > > Yes, 2.8.2 and 2.9.2 will fix the problem.
> > >
> > > Penghui
> > >
> > > On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> > > massimilianomirelli...@gmail.com> wrote:
> > >
> > > > Thank you for the rc!
> > > >
> > > > +1 (non-binding)
> > > >
> > > > * verify sha512 checksums
> > > > * verify gpg signatures
> > > > * build pulsar-all docker image
> > > > * execute Fallout distributed system test (produce / receive 10k
> > > messages)
> > > >
> > > > Building the docker image as indicated in the README:
> > > >
> > > > mvn clean install -DskipTests
> > > > mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
> > > >
> > > > I still get the error described in this PR#11951 (
> > > > https://github.com/apache/pulsar/pull/11951) which I suppose has
> been
> > > > cherry picked in 2.9.1.
> > > >
> > > > Giving enough permissions to /docker/pulsar/scripts/ in the src
> package
> > > and
> > > > then building the docker image again solved the issue.
> > > >
> > > > I also tested the problem with the image Enrico provided, this way:
> > > >
> > > > docker run -it --entrypoint bash eolivelli/pulsar-all:2.9.1rc2
> > > > ls -al /pulsar/bin
> > > >
> > > > and that one does have the right permissions.
> > > >
> > > > Enrico, did you build the image using the same mvn commands (^^^), or
> > is
> > > > there some other way to build it?
> > > >
> > > > Thank you,
> > > > Max
> > > >
> > > > On Fri, 17 Dec 2021 at 16:18, 陳智弘  wrote:
> > > >
> > > > > Hi PengHu,
> > > > >
> > > > >  Will this issue be fixed in the future releases?
> > > > >
> > > > > PengHui Li  於 2021年12月17日 週五 21:53 寫道：
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > I'm ok, it only happens when the message is without a schema
> > version.
> > > > > > So I'm not giving -1.
> > > > > >
> > > > > > Thanks,
> > > > > > Penghui
> > > > > >
> > > > > >
> > > > > > On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli <
> > eolive...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Peng Hui,
> > > > > > >
> > > > > > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li <
> > > > > peng...@apache.org>
> > > > > > > ha
> > > > > > > scritto:
> > > > > > >
> > > > > > > > Checked:
> > > > > > > >
> > > > > > > > - Build from the src
> > > > > > > > - Check signatures
> > > > > > > > - Follow the validation process
> > > > > > > >
> > > > > > > > But when I try to verify PulsarSQL, got following exceptions:
> > > > > > > >
> > > > > > > > ```
> > > > > > > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3
> > > > > > > > io.prestosql.execution.StageStateMachine Stage
> > > > > > > > 20211217_065818_1_cahiv.1 failed
> > > > > > > >
> com.google.common.util.concurrent.UncheckedExecutionException:
> > > > > > > > java.nio.BufferUnderflowException
> > 

Re: Status of Pulsar 2.9.0 and starting 2.9.1

[Enrico Olivelli]

Yu,

Il giorno lun 20 dic 2021 alle ore 10:01 Yu  ha scritto:

> Hi Enrico,
>
> Thanks for your contribution.
>
> 1) A soft reminder that is not on the current release process [a]:
> Since 2.9.0 was delayed, some doc changes on the master are applied to
> 2.10.0 only, so generating the 2.9.0 doc set should be based on the "2.9.0
> release time point" rather than the current master, or else the 2.9.0 doc
> set might contain 2.10 feature docs.
>

This is actually a good point.
I am not sure about how I can fix this.
Maybe I can  try to create the versioned docs from the v2.9.0 tag



>
> 2) 2.9.0 is not announced and the 2.9.0 release note is available on the
> website.
> For the previous releases, we have tech blogs for them to (announce and)
> explain highlights [b], will you create a tech blog for 2.9.0?
>

I am sorry but I won't have time to do this before the beginning of January,
it would be useful if someone could do it

Enrico



>
> [a]
> https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
> [b] https://pulsar.apache.org/blog/2021/09/23/Apache-Pulsar-2-8-1/
>
>
> On Fri, Dec 17, 2021 at 7:36 PM Enrico Olivelli 
> wrote:
>
> > Il giorno ven 17 dic 2021 alle ore 09:45 Yu  ha
> scritto:
> >
> > > Hi Enrico,
> > >
> > > Thanks for your great effort on the 2.9.0 release.
> > >
> > > Circling back to see if there is any progress of the 2.9.0 website
> > updates
> > > [1].
> > >
> > > Currently, the 2.9.0 doc is not available on the website and the 2.9.0
> > doc
> > > set has not been generated yet, any updates? Thanks
> > >
> >
> >
> > I will do it together with 2.9.1
> > because 2.9.0 is broken, so it is better to not do much buzz around it
> >
> > Enrico
> >
> >
> > >
> > > [1]
> > >
> https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site
> > >
> > >
> > > On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli 
> > > wrote:
> > >
> > > > Dave,
> > > > You are correct.
> > > > Pulsar 2.9.0 has already been released and also some people already
> > > started
> > > > to report issues.
> > > > The docker images have been deployed and we cannot change them.
> > > >
> > > > I am finishing the release process for 2.9.0 with the website
> updates.
> > > >
> > > > I am preparing 2.9.1.
> > > >
> > > > I propose to just skip the announcement for 2.9.0.
> > > >
> > > > If we are quick during the VOTE we can close this story within the
> end
> > of
> > > > the week
> > > >
> > > > Enrico
> > > >
> > > > Il Lun 13 Dic 2021, 06:34 Dave Fisher  ha
> > > scritto:
> > > >
> > > > > (1) we have published 2.9.0 at
> > > > > https://downloads.apache.org/pulsar/pulsar-2.9.0/
> > > > >
> > > > > (2) we have published 2.9.0 artifacts through maven central. They
> > don’t
> > > > > let anyone republish versions.
> > > > >
> > > > > There are no do overs on versions. We simply cannot redo 2.9.0 at
> > this
> > > > > moment.
> > > > >
> > > > > All the best,
> > > > > Dave
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo 
> wrote:
> > > > > >
> > > > > > My take is - if we haven't announced 2.9, I would suggest just
> > > redoing
> > > > > the
> > > > > > 2.9.0 release.
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen 
> > > > wrote:
> > > > > >>
> > > > > >> I am a little confused about why we should skip 2.9.0 and not
> > > continue
> > > > > >> to release 2.9.0 by including the critical bug fixes. In fact,
> the
> > > > > >> Pulsar 2.9.0 release is not yet completed.
> > > > > >>
> > > > > >> For users, they will worry about whether the Pulsar release
> > process
> > > is
> > > > > >> standardized if we skip 2.9.0. They will also worry about the
> > > release
> > > > > >> quality of Apache Pulsar if 

Re: [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

Il giorno lun 20 dic 2021 alle ore 13:21 PengHui Li  ha
scritto:

> Hi Enrico,
>
> Have you checked the CI status after completing the 2.9.1 PRs
> cherry-picking?
>

When I created the tag I am pretty sure that CI on GH actions passed.

I hope that no-one committed something to branch-2.9

If this happens, then we must enforce some rules about the fact that only
the RM can commit to a branch under release.

In ZooKeeper we create a "release branch" in order to prevent such problems

Enrico



>
> Looks this test failed when I run the test with tag 2.9.1.
>
> ```
> [INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time elapsed:
> 26.782 s <<< FAILURE! - in
> org.apache.pulsar.broker.transaction.TransactionTest
> [ERROR]
>
> testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>  Time elapsed: 0.107 s  <<< FAILURE!
> java.lang.AssertionError: expected [4] but found [3]
> at org.testng.Assert.fail(Assert.java:99)
> at org.testng.Assert.failNotEquals(Assert.java:1037)
> at org.testng.Assert.assertEqualsImpl(Assert.java:140)
> at org.testng.Assert.assertEquals(Assert.java:122)
> at org.testng.Assert.assertEquals(Assert.java:907)
> at org.testng.Assert.assertEquals(Assert.java:917)
> at
>
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
> at
>
> org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
> at
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
> at
> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]
>
> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
> [INFO]   Run 1: PASS
> [ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
> expected [4] but found [3]
> [INFO]
> [INFO]
> [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
> ```
>
> Regards,
> Penghui
>
> On Mon, Dec 20, 2021 at 3:06 PM Enrico Olivelli 
> wrote:
>
> > Hello everyone,
> > The VOTE passed with 5 VOTEs, 3 of them were binding.
> >
> > - Matteo Merli (binding)
> > - Enrico Olivelli (binding)
> > - Peng Hui (binding)
> > - Nicolò Boschi
> > - Massimiliano Mirelli
> >
> > I will move forward with the next steps
> > https://github.com/apache/pulsar/wiki/Release-process
> >
> > Enrico
> >
> >
> > Il giorno sab 18 dic 2021 alle ore 11:41 PengHui Li 
> > ha
> > scritto:
> >
> > > +1 binding
> > >
> > > Penghui
> > >
> > > Enrico Olivelli 于2021年12月18日 周六18:39写道：
> > >
> > > > +1 (binding)
> > > >
> > > > - Run release validation procedure
> > > > - CI is passing on those sources
> > > >
> > > > Enrico
> > > >
> > > > Il giorno sab 18 dic 2021 alle ore 02:51 PengHui Li <
> > peng...@apache.org>
> > > > ha
> > > > scritto:
> > > >
> > > > > >  Will this issue be fixed in the future releases?
> > > > >
> > > > > Yes, 2.8.2 and 2.9.2 will fix the problem.
> > > > >
> > > > > Penghui
> > > > >
> > > > > On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli <
> > > > > massimilianomirelli...@gmail.com> wrote:
> > > > >
> > > > > > Thank you for the rc!
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > * verify sha512 checksums
> > > > > > * verify gpg signatures
> > > > > > * build pulsar-all docker image
>

DISCUSS - [RESULT] [VOTE] Apache Pulsar 2.9.1 candidate 2

[Enrico Olivelli]

(changing the subject)


Il giorno lun 20 dic 2021 alle ore 13:33 PengHui Li  ha
scritto:

> I use the 2.9.1 tag to run the test, not branch-2.9.
>
> I think this one(2db23b8dd3eb859cc16e30686578be275026c347) is the last one
> that you cherry-picked?
>

This is the latest commit: f52ac045f41acbb6c31da21a3463df3cfbe8f1b4

And you are correct, this is the latest commit I cherry-picked:
2db23b8dd3eb859cc16e30686578be275026c347 [Managed Ledger] Fix the incorrect
total size when BrokerEntryMetadata is enabled (#12714)

for reference:
 https://github.com/apache/pulsar/tree/v2.9.1

commit f52ac045f41acbb6c31da21a3463df3cfbe8f1b4 (HEAD -> branch-2.9, tag:
v2.9.1-candidate-2, tag: v2.9.1, origin/branch-2.9)
Author: Nicolò Boschi 
Date:   Wed Dec 15 22:18:58 2021 +0100
[security] Upgrade Netty to 4.1.72 - CVE-2021-43797 (#13328)


Enrico


But there are failed tests:
>
> [image: image.png]
>
> Penghui
>
> On Mon, Dec 20, 2021 at 8:24 PM Enrico Olivelli 
> wrote:
>
>> Il giorno lun 20 dic 2021 alle ore 13:21 PengHui Li 
>> ha
>> scritto:
>>
>> > Hi Enrico,
>> >
>> > Have you checked the CI status after completing the 2.9.1 PRs
>> > cherry-picking?
>> >
>>
>> When I created the tag I am pretty sure that CI on GH actions passed.
>>
>> I hope that no-one committed something to branch-2.9
>>
>> If this happens, then we must enforce some rules about the fact that only
>> the RM can commit to a branch under release.
>>
>> In ZooKeeper we create a "release branch" in order to prevent such
>> problems
>>
>> Enrico
>>
>>
>>
>> >
>> > Looks this test failed when I run the test with tag 2.9.1.
>> >
>> > ```
>> > [INFO] Running org.apache.pulsar.broker.transaction.TransactionTest
>> > [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 12, Time
>> elapsed:
>> > 26.782 s <<< FAILURE! - in
>> > org.apache.pulsar.broker.transaction.TransactionTest
>> > [ERROR]
>> >
>> >
>> testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>> >  Time elapsed: 0.107 s  <<< FAILURE!
>> > java.lang.AssertionError: expected [4] but found [3]
>> > at org.testng.Assert.fail(Assert.java:99)
>> > at org.testng.Assert.failNotEquals(Assert.java:1037)
>> > at org.testng.Assert.assertEqualsImpl(Assert.java:140)
>> > at org.testng.Assert.assertEquals(Assert.java:122)
>> > at org.testng.Assert.assertEquals(Assert.java:907)
>> > at org.testng.Assert.assertEquals(Assert.java:917)
>> > at
>> >
>> >
>> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(TransactionTest.java:145)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:498)
>> > at
>> >
>> >
>> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
>> > at
>> >
>> >
>> org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
>> > at
>> >
>> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
>> > at
>> >
>> org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> > [INFO]
>> > [INFO] Results:
>> > [INFO]
>> > [ERROR] Failures:
>> > [ERROR]
>> >
>> >
>> org.apache.pulsar.broker.transaction.TransactionTest.testCreateTransactionSystemTopic(org.apache.pulsar.broker.transaction.TransactionTest)
>> > [INFO]   Run 1: PASS
>> > [ERROR]   Run 2: TransactionTest.testCreateTransactionSystemTopic:145
>> > expected [4] but found [3]
>> > [INFO]
>> > [INFO]
>> > [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 7
>> > ```
>> >
>> > Regards,
>> > Penghui
>> >
>> &

OSX packages status....broken ?!?

[Enrico Olivelli]

Hello,
According to the release procedure I am trying to build the OSX packages
but the build fails.

This is the issue:
https://github.com/apache/pulsar/issues/13429

Apart from fixing the problem itself, I wonder if it would be better to
create a CI job that periodically runs such a procedure.

The net result is:
- the procedure does not work
- even if it worked, there is no guarantee that the packages work well

In the meantime I will proceed to the next step in the release process for
2.9.1

Thoughts ?


Enrico

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

[Enrico Olivelli]

Il giorno mar 21 dic 2021 alle ore 11:49 Shivji Kumar Jha <
shiv4...@gmail.com> ha scritto:

> Hi LinLin,
>
> Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we move
> to Log4j 2.17.0 in 2.8.2?
>

is it included
https://github.com/apache/pulsar/tree/v2.8.2-candidate-2

Enrico


>
> Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did not
> > protect from uncontrolled recursion from self-referential lookups. This
> > allows an attacker with control over Thread Context Map data to cause a
> > denial of service when a crafted string is interpreted. This issue was
> > fixed in Log4j 2.17.0 and 2.12.3.
>
>
>
> [1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Tue, 21 Dec 2021 at 14:42, Masahiro Sakamoto 
> wrote:
>
> > +1 (binding)
> >
> > - Checked checksums and signatures
> > - Checked license headers using Apache Rat
> > - Compiled the source
> > - Ran the standalone server
> > - Confirmed that producer and consumer work properly
> > - Validated functions, connectors, and stateful functions
> >
> > Regards,
> >
> > Masahiro Sakamoto
> > Yahoo Japan Corp.
> > E-mail: massa...@yahoo-corp.jp
> >
> > -Original Message-
> > From: Hiroyuki Sakai 
> > Sent: Tuesday, December 21, 2021 1:07 PM
> > To: dev@pulsar.apache.org
> > Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2
> >
> > +1 (binding)
> >
> >  - check signatures/checksums
> >  - Built sources
> >  - Validate Pub/Sub and Java Functions
> >  - Validate Connectors
> >  - Validate Stateful Functions
> >
> > Regards,
> > Hiroyuki
> >
> >
> > 
> > From: linlin 
> > Sent: Monday, December 20, 2021 20:18
> > To: dev@pulsar.apache.org 
> > Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2
> >
> > This is the second release candidate for Apache Pulsar, version 2.8.2
> >
> > It fixes the following issues:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> > Source and binary files:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0
> >
> > SHA-512 checksums:
> >
> >
> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
> >  apache-pulsar-2.8.2-bin.tar.gz
> >
> >
> >
> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
> >  apache-pulsar-2.8.2-src.tar.gz
> >
> > Maven staging repo:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1129%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4TVSZI76N8xalmLLwJyvtUJkkpNC7T0s2Taj%2Fc5CKgg%3D&reserved=0
> >
> > The tag to be voted upon:
> > v2.8.2-candidate-2 (4b9cadcd57e41bc8eb95cc9b9917f938365b1cca)
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Freleases%2Ftag%2Fv2.8.2-candidate-2&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Ba6OsrGNvcCsuYFgPp%2Btg49yRHK%2FVBmWumN7haK%2Birw%3D&reserved=0
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >
> >
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ

Re: [VOTE] Pulsar Release 2.7.4 Candidate 2

[Enrico Olivelli]

+1 (binding)

- built from sources, JDK8 in MacOS
- run pulsar standalone, smoke tests
- verified checksum and digital signatures
- I took a look to the Maven staging repository

Thanks for driving the release

Enrico


Il giorno mar 21 dic 2021 alle ore 11:27 Nozomi Kurihara <
nkuri...@apache.org> ha scritto:

> +1 (binding)
>
> - Verified checksums and signatures
> - Verified license headers by apache rat
> - Built the source
> - Ran standalone, producer, and consumer
> - Verified functions, cassandra connectors and stateful functions
>
> Thanks for driving the release.
> Nozomi
>
> 2021年12月21日(火) 18:34 guo jiwei :
>
> > This is the second release candidate for Apache Pulsar, version 2.7.4.
> >
> > It fixes the following issues:
> >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Aopen+label%3Arelease%2F2.7.4+is%3Apr+
> >
> > Release note:
> > https://github.com/apache/pulsar/pull/13391
> >
> > *** Please download, test and vote on this release. This vote will
> > stay openfor at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.4-candidate-2/
> >  >
> >
> > SHA-512 checksums:
> >
> >
> >
> a56a794bd14c8f4b6d7320496b2be4640a836a7aa940c37aac4f53ae057f8b4887f3f8716f475d2a03c79e3e44bf2ad9234fd05e2b0f58b005a0a4f7f376fc15
> >  apache-pulsar-2.7.4-bin.tar.gz
> >
> >
> 03ec488d59d281fdbbbee48f1f62b266cd1163181decbf962d0d58eacc746731d31739955d8afac8bc392547caf2ae76e864d0ad2f2a460cd53920a707a7b40c
> >  apache-pulsar-2.7.4-src.tar.gz
> >
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1
> >  > >130
> >
> > The tag to be voted upon:
> > v2.7.4-candidate-2 (2774b464f3b7e1ccfdc84d6d390efe7260cbb7fa)
> > https://github.com/apache/pulsar/releases/tag/v2.7.4-candidate-
> > 2
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
>

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

[Enrico Olivelli]

+1 (binding)

- built from sources, JDK8 in MacOS
- run pulsar standalone, smoke tests
- verified checksum and digital signatures
- I took a look to the Maven staging repository (verified that "integration
tests jars" are present)

Thanks for driving the release

Enrico

Il giorno mar 21 dic 2021 alle ore 11:54 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

>
>
> Il giorno mar 21 dic 2021 alle ore 11:49 Shivji Kumar Jha <
> shiv4...@gmail.com> ha scritto:
>
>> Hi LinLin,
>>
>> Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we
>> move
>> to Log4j 2.17.0 in 2.8.2?
>>
>
> is it included
> https://github.com/apache/pulsar/tree/v2.8.2-candidate-2
>
> Enrico
>
>
>>
>> Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did
>> not
>> > protect from uncontrolled recursion from self-referential lookups. This
>> > allows an attacker with control over Thread Context Map data to cause a
>> > denial of service when a crafted string is interpreted. This issue was
>> > fixed in Log4j 2.17.0 and 2.12.3.
>>
>>
>>
>> [1] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
>>
>> Regards,
>> Shivji Kumar Jha
>> http://www.shivjijha.com/
>> +91 8884075512
>>
>>
>> On Tue, 21 Dec 2021 at 14:42, Masahiro Sakamoto 
>> wrote:
>>
>> > +1 (binding)
>> >
>> > - Checked checksums and signatures
>> > - Checked license headers using Apache Rat
>> > - Compiled the source
>> > - Ran the standalone server
>> > - Confirmed that producer and consumer work properly
>> > - Validated functions, connectors, and stateful functions
>> >
>> > Regards,
>> >
>> > Masahiro Sakamoto
>> > Yahoo Japan Corp.
>> > E-mail: massa...@yahoo-corp.jp
>> >
>> > -Original Message-
>> > From: Hiroyuki Sakai 
>> > Sent: Tuesday, December 21, 2021 1:07 PM
>> > To: dev@pulsar.apache.org
>> > Subject: Re: [VOTE] Apache Pulsar 2.8.2 candidate 2
>> >
>> > +1 (binding)
>> >
>> >  - check signatures/checksums
>> >  - Built sources
>> >  - Validate Pub/Sub and Java Functions
>> >  - Validate Connectors
>> >  - Validate Stateful Functions
>> >
>> > Regards,
>> > Hiroyuki
>> >
>> >
>> > 
>> > From: linlin 
>> > Sent: Monday, December 20, 2021 20:18
>> > To: dev@pulsar.apache.org 
>> > Subject: [VOTE] Apache Pulsar 2.8.2 candidate 2
>> >
>> > This is the second release candidate for Apache Pulsar, version 2.8.2
>> >
>> > It fixes the following issues:
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fissues%3Fq%3Dlabel%253Acherry-picked%252Fbranch-2.8%2Blabel%253Arelease%252F2.8.2%2Bis%253Aclosed&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qYRCQ3Fohdy%2B6U2trDKNTO0w406ATKPZ6cPZExBQTJ4%3D&reserved=0
>> >
>> > *** Please download, test and vote on this release. This vote will stay
>> > open
>> > for at least 72 hours ***
>> >
>> > Note that we are voting upon the source (tag), binaries are provided for
>> > convenience.
>> > Source and binary files:
>> >
>> >
>> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.8.2-candidate-2%2F&data=04%7C01%7Cmassakam%40yahoo-corp.jp%7Cb6c6039ab7634eb56b9208d9c43769e6%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637756565724741012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TfezF%2BDzswX5xpbXNM1GXzW8Msqj1GkjGt4ULF6xKZA%3D&reserved=0
>> >
>> > SHA-512 checksums:
>> >
>> >
>> 59aa0a14188a766ce802ba30cbaa2584a1904d26d8f41f164d3f97ea1970aa1391e11755d8077c96aeb136d2b9d73bf8b768978de7fa7f71d69cb57e2c1fce8c
>> >  apache-pulsar-2.8.2-bin.tar.gz
>> >
>> >
>> >
>> 82a1423fda4004297ca2867279077ed261c7149be96deca2c127ba5b91af08fec80dc4a8b15ee2ba8704e209efa577a0c7b4cfb78341d3a43a38bf476c894c5c
>> >  apache-pulsar-2.8.2-src.tar.gz
>> >
>> > Maven staging repo:
>> >
>> >
>> https://jpn01.safelinks.protect

WebSite build CI job does work - blocker for announcing release !

[Enrico Olivelli]

Hello,
the WebSite build CI job is stuck
https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true

This is probably the problem, it is due to "crowdin" that is timing out

File 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
SKIPPED
File 'master/website/i18n/en.json'/ - - OK
Done in 2452.38s.
+ yarn run crowdin-download
yarn run v1.22.5
warning package.json: No license field
$ crowdin --config ../crowdin.yaml download -b master
Error: The operation was canceled.


Can anyone help ?
we are stuck and if we do not update the website we cannot announce the
releases with the security fixes


Enrico

Re: [PR] Pulsar non root docker image

[Enrico Olivelli]

Michael,

I would like to add this item to the list
https://github.com/apache/pulsar/pull/10815

Basically if you are running as non root, you cannot add tools on demand,
so we need to add basic tools, like netstat/vim/unzip otherwise when
you have a problem you are trapped.

there are ways to inject tools, but they are very hard to execute for
people who is not very experienced

Enrico


Il giorno mer 22 dic 2021 alle ore 04:40 Haiting Jiang <
jianghait...@apache.org> ha scritto:

> > 1. Pulsar configuration is read in only from configuration files in
> > `/pulsar/conf`. A non root user must be able to update these files to
> > have run with custom configuration.
>
> About the configurations, I also see similar require like this lately [0].
> IMHO, update any configs without redeploy service is a useful feature.
> I would like post a PIP for this later.
> Basic idea is like make all configs dynamic by default except
> `metadataStoreUrl` and all configs are stored under path
> "/admin/configuration" in metadata store.
>
> [0] https://github.com/apache/pulsar/pull/13074
>
> On 2021/12/21 20:16:44 Michael Marshall wrote:
> > All tests are now passing for this PR [0]. I built the docker image
> > and pushed it to my personal repository to simplify testing [1] for
> > anyone interested in verifying the changes.
> >
> > I would like our docker image to be as close to immutable as possible.
> > As far as I can tell, here are the only reasons the image cannot be
> > immutable:
> >
> > 1. Pulsar configuration is read in only from configuration files in
> > `/pulsar/conf`. A non root user must be able to update these files to
> > have run with custom configuration.
> > 2. The Pulsar function worker unpacks nar files to
> > `/pulsar/download/pulsar_functions` by default.
> > 3. Pulsar tiered storage writes to `/pulsar` by default when using
> > filesystem storage.
> > 4. The Presto SQL worker writes to `/pulsar/lib/presto/` by default.
> > 5. Pulsar-admin and functions write to `/pulsar/log` by default
> > (possibly other components too).
> >
> > Note that even though bookkeepers and zookeepers write to
> > `/pulsar/data`, they should be writing to docker volumes, in which
> > case, the host's file system permissions are all that matter.
> >
> > If we can remove any of the above reasons, we can decrease the
> > privilege in the docker image.
> >
> > The PR has more detail. Please take a look.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/13376
> > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT-1dec8b9
> >
> >
> > On Fri, Dec 17, 2021 at 12:33 AM Michael Marshall 
> wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > I opened a PR to make our pulsar and pulsar-all docker images non root
> > > and OpenShift compliant [0]. As some may remember, we had issues with
> > > these changes before due to lack of testing. I plan to test thoroughly
> > > before we merge this PR, and it'd be great to have others test too. I
> > > published a build of my PR [1]. I also have an issue [2] tracking this
> > > work.
> > >
> > > Please take a look. I hope to make our 2.10 release a non root release!
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/13376
> > > [1] michaelmarshall/pulsar:2.10.0-SNAPSHOT
> > > [2] https://github.com/apache/pulsar/issues/11269
> >
>

Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool

[Enrico Olivelli]

very good

Enrico

Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
mattisonc...@gmail.com> ha scritto:

> +1
>
> On Wed, 22 Dec 2021 at 07:59, Matteo Merli  wrote:
>
> > This is the voting thread for PIP-123. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13346
> >
> >
> > -
> > ## Motivation
> >
> > For a very long time, we have included a CLI command to start the
> ZooKeeper
> > shell utility: `pulsar zookeeper-shell`, which is essentially a
> repackaging
> > of the ZooKeeper tool `zkCli.sh`.
> >
> > This is useful in some cases to either verify the content of metadata or
> > to perform cleanup and modification tasks for which there is not an
> > available
> > option through the Pulsar REST APIs.
> >
> > While it's very useful, there are some drawbacks with the
> `zookeeper-shell`
> > as it is today:
> >
> >  1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > metadata backends, we should have a tool that works across all the
> > implementations and presents a single consistent interface.
> >
> >  2. ZooKeeper shell is designed to be an interactive shell and it's not
> > very
> > good when trying to do non-interactive scriptable operations.
> >
> >  3. ZooKeeper is a bit clunky when using it and it requires the user to
> > retype
> > paths many times. The commands are not very intuitive or documented.
> > It's not possible to update z-node with multi-lines content.
> >
> >  4. We cannot easily add functionality or improvements into ZooKeeper
> > shell,
> > since it belongs to a different project and the tool has been
> > stagnating
> > for many years.
> >
> >  5. In cases where the z-nodes content is binary (Protobuf) or
> compressed,
> > there
> > is no easy way to inspect the content from the ZooKeeper shell.
> > Additionally, we can format and colorize JSON content to make it
> > easier to
> > read.
> >
> >  6. The paths used for metadata resources are also often using encodings
> > that
> > make it more difficult to construct on the shell tool.
> >
> > Part of what is described here is in the `pulsar-managed-ledger-admin`
> CLI
> > tool,
> > though that is a Python script that requires additional dependencies that
> > are
> > not typically installed, it only works with ZooKeeper, and it only
> targets
> > accessing the managed ledger metadata.
> >
> > ## Goal
> >
> > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > solves
> > all the issues described above.
> >
> > We would leave the `zookeeper-shell` command for now. In the future, once
> > the
> > new tool is proven, we can consider removing the `zookeeper-shell`
> command.
> >
> >
> > ## Proposed changes
> >
> > Add a new command:
> > ```bash
> > bin/pulsar metadata
> > ```
> >
> > with several subcommands:
> >
> >
> >  Get
> >
> > Examples:
> > ```bash
> > # General path get
> > $ pulsar metadata get /my-path
> >
> >
> > # Topic metadata
> > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > # Namespace get
> > $ pulsar metadata get namespace my-tenant/my-namespace
> > {
> >   # Namespace metadata
> > }
> >
> > $ pulsar metadata get ledger 12345
> > {
> >   # BK ledger metadata
> > }
> > ```
> >
> >  Delete
> >
> > Examples:
> > ```bash
> > # General path delete
> > $ pulsar metadata delete /my-path
> >
> > # Topic metadata
> > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > ```
> >
> >  Scan
> >
> > Examples:
> > ```bash
> > $ pulsar metadata scan /my-path
> > /my-path
> > /my-path/1
> > /my-path/2
> > /my-path/3
> > /my-path/3/1
> >
> > $ pulsar metadata scan --values /my-path
> > /my-path
> > {value}
> >
> > /my-path/1
> > {value}
> >
> > /my-path/2
> > {value}
> > ```
> >
> >  Shell
> >
> > ```bash
> > $ pulsar metadata shell
> > > get topic my-tenant/my-namespace/my-topic
> > {
> >   # Managed ledger metadata
> > }
> >
> > > delete topic my-tenant/my-namespace/my-topic
> >
> > > cd /my-path
> > > ls
> > 1
> > 2
> > 3
> > > delete 1 # Delete keys with relative paths
> >
> > ```
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

[Enrico Olivelli]

Lari,

Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari 
ha scritto:

> Dear Pulsar community members,
>
> PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
> repositories" was created in April 2020. The repositories for
> pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
> ago [2].
>
> I'd like to propose that we continue with the PIP-62 plan asap, before
> Apache Pulsar 2.10.0 is released.
>
> I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> waiting for the Trino Pulsar PR [3] being completed.
>
> I am volunteering to making the changes as per the PIP-62 plan.
> When can we proceed with the plan? Do we need an explicit vote on the
> mailing list about this?
>

We discussed this many times in the past year.
I believe that there is no need to wait

Enrico



>
> BR,
>
> Lari
>
> [1]
>
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> [2]
>
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> [3] https://github.com/trinodb/trino/pull/8020
>
> Other email threads:
> * [Discuss] Don't include presto/trino in the normal Pulsar distribution -
> https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> * Updates on Presto connector for PIP-62 -
> https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[Enrico Olivelli]

1 (binding)

Enrico

Il giorno mer 22 dic 2021 alle ore 03:39 mattison chao <
mattisonc...@gmail.com> ha scritto:

> +1
>
> On Wed, 22 Dec 2021 at 07:46, Matteo Merli  wrote:
>
> > This is the voting thread for PIP-120. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13306
> >
> > 
> >
> > ## Motivation
> >
> > In Pulsar 2.8, we have introduced a setting to control the amount of
> memory
> > used by a client instance.
> >
> > ```java
> > interface ClientBuilder {
> > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > }
> > ```
> >
> > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> > being
> > enforced.
> >
> > I think it's a good time for 2.10 to enable this setting by default and,
> > correspondingly, to disable by default the producer queue size limit.
> >
> > This will simplify a lot the configuration that a producer application
> will
> > have to come up with, when publishing with many topic/partitions or
> > when messages
> > are bigger than expected.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the `ClientBuilder`, change
> >   * `memoryLimit`: 0 -> 64 MB
> >
> > For the `ProducerBuilder`, changes
> >   * `maxPendingMessages`: 1000 -> 0
> >
> > 64MB is picked because it's a small enough memory size that will
> guarantee
> > a very high producer throughput, irrespective of the individual messages
> > size.
> >
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: WebSite build CI job does work - blocker for announcing release !

[Enrico Olivelli]

Thanks Dave,
I don't see 2.9.1 in the download page.
Did I miss to update some files?

Enrico

Il Mer 22 Dic 2021, 21:19 Dave Fisher  ha scritto:

> Hi -
>
> A WebSite Build CI Job just completed. Please check that it created the
> new docs.
>
> I see the release notes for 2.9.1 -
> https://pulsar.apache.org/en/release-notes/
>
> Regards,
> Dave
>
> > On Dec 21, 2021, at 11:33 PM, Nicolò Boschi 
> wrote:
> >
> > I think the problem is that it times out (currently there's a job timeout
> > of 2h)
> > I created this PR to increase it to 3h
> > https://github.com/apache/pulsar/pull/13449
> >
> >
> > Il giorno mer 22 dic 2021 alle ore 08:23 Enrico Olivelli <
> > eolive...@gmail.com> ha scritto:
> >
> >> Hello,
> >> the WebSite build CI job is stuck
> >> https://github.com/apache/pulsar/runs/4601374159?check_suite_focus=true
> >>
> >> This is probably the problem, it is due to "crowdin" that is timing out
> >>
> >> File
> 'master/website/versioned_docs/version-2.6.3/concepts-overview.md'| -
> >> SKIPPED
> >> File 'master/website/i18n/en.json'/ - - OK
> >> Done in 2452.38s.
> >> + yarn run crowdin-download
> >> yarn run v1.22.5
> >> warning package.json: No license field
> >> $ crowdin --config ../crowdin.yaml download -b master
> >> Error: The operation was canceled.
> >>
> >>
> >> Can anyone help ?
> >> we are stuck and if we do not update the website we cannot announce the
> >> releases with the security fixes
> >>
> >>
> >> Enrico
> >>
> >
> >
> > --
> > Nicolò Boschi
>
>

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 16:59 Michael Marshall  ha
scritto:

> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:09 AM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > PengHui Li  于2021年12月22日周三 19:33写道：
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Wed, Dec 22, 2021 at 7:23 AM Matteo Merli 
> wrote:
> > >
> > > > This is the voting thread for PIP-119. It will stay open for at
> least 48h.
> > > >
> > > > https://github.com/apache/pulsar/issues/13305
> > > >
> > > > ---
> > > >
> > > > ## Motivation
> > > >
> > > > The consistent hashing implementation to uniformly assign keys to
> consumers
> > > > in the context of a KeyShared subscription, was introduced in
> > > > https://github.com/apache/pulsar/pull/6791, which was released in
> Pulsar
> > > > 2.6.0.
> > > >
> > > > While consistent hashing can use slightly more memory in certain
> cases, it
> > > > is
> > > > more suitable as a general default implementation, as it leads to a
> fairer
> > > > distribution of keys across consumers, and avoiding corner cases that
> > > > depend
> > > > on the sequence of addition/removal of consumers.
> > > >
> > > > ## Proposed changes
> > > >
> > > > In 2.10 release, for the setting:
> > > >
> > > > ```properties
> > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > > splitting ranges or
> > > > # consistent hashing to reassign keys to new consumers
> > > > subscriptionKeySharedUseConsistentHashing=false
> > > > ```
> > > >
> > > > Change its default value to `true`.
> > > >
> > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will
> still
> > > > be
> > > > able to use the old implementation.
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >
>

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 13:09 Hang Chen  ha scritto:

> +1
>
> Thanks,
> Hang
>
> PengHui Li  于2021年12月22日周三 19:34写道：
> >
> > +1
> >
> > Penghui
> >
> > On Wed, Dec 22, 2021 at 7:22 AM Matteo Merli  wrote:
> >
> > > https://github.com/apache/pulsar/issues/13304
> > >
> > > Following the discussion, I have updated the proposal to also include
> > > the deprecation and renaming of the config setting name to
> > > `metadataSessionExpiredPolicy`.
> > >
> > >
> > >
> > > ---
> > > ## Motivation
> > >
> > > After all the work done for PIP-45 that was already included in 2.8
> and 2.9
> > > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > > election.
> > >
> > > Another important change was to avoid doing any deferrable metadata
> > > operation
> > > when we know that we are not currently connected to the metadata
> service.
> > >
> > > Finally, that enabled to stabilize in 2.9 the configuration setting
> that
> > > allows
> > > brokers to continue operating in a safe mode when the session with
> > > ZooKeeper
> > > expires.
> > >
> > > The way it works is that, when we lose a ZooKeeper session, the data
> plane
> > > will
> > > continue to work undisturbed, relying on the BookKeeper fencing to
> avoid
> > > any
> > > inconsistencies.
> > >
> > > New topics are not able to get started, but existing topics will see no
> > > impact.
> > >
> > > The original intention for shutting down the brokers was to ensure
> that we
> > > would automatically go back to a consistent state, with respect to
> which
> > > resources are "owned" in ZooKeeper by a given broker.
> > >
> > > With the re-acquirable resource locks, that problem was solved and
> > > thoroughly
> > > tested to be robust.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the setting:
> > >
> > > ```properties
> > > # There are two policies to apply when broker metadata session
> > > expires: session expired happens, "shutdown" or "reconnect".
> > > # With "shutdown", the broker will be restarted.
> > > # With "reconnect", the broker will keep serving the topics, while
> > > attempting to recreate a new session.
> > > zookeeperSessionExpiredPolicy=shutdown
> > > ```
> > >
> > > Deprecate the old setting and rename it to
> > > `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
> > >
>

Re: [VOTE] PIP-120: Enable client memory limit by default

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mer 22 Dic 2021, 16:58 Michael Marshall  ha
scritto:

> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:10 AM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > PengHui Li  于2021年12月22日周三 19:28写道：
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Wed, Dec 22, 2021 at 6:57 PM ZhangJian He 
> wrote:
> > >
> > > > +1
> > > >
> > > > Christophe Bornet  于2021年12月22日周三 17:38写道：
> > > >
> > > > > +1
> > > > >
> > > > > Le mer. 22 déc. 2021 à 00:46, Matteo Merli  a
> écrit :
> > > > >
> > > > > > This is the voting thread for PIP-120. It will stay open for at
> least
> > > > > 48h.
> > > > > >
> > > > > > https://github.com/apache/pulsar/issues/13306
> > > > > >
> > > > > > 
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > In Pulsar 2.8, we have introduced a setting to control the
> amount of
> > > > > memory
> > > > > > used by a client instance.
> > > > > >
> > > > > > ```java
> > > > > > interface ClientBuilder {
> > > > > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no
> limit
> > > > is
> > > > > > being
> > > > > > enforced.
> > > > > >
> > > > > > I think it's a good time for 2.10 to enable this setting by
> default
> > > > and,
> > > > > > correspondingly, to disable by default the producer queue size
> limit.
> > > > > >
> > > > > > This will simplify a lot the configuration that a producer
> application
> > > > > will
> > > > > > have to come up with, when publishing with many topic/partitions
> or
> > > > > > when messages
> > > > > > are bigger than expected.
> > > > > >
> > > > > > ## Proposed changes
> > > > > >
> > > > > > In 2.10 release, for the `ClientBuilder`, change
> > > > > >   * `memoryLimit`: 0 -> 64 MB
> > > > > >
> > > > > > For the `ProducerBuilder`, changes
> > > > > >   * `maxPendingMessages`: 1000 -> 0
> > > > > >
> > > > > > 64MB is picked because it's a small enough memory size that will
> > > > > guarantee
> > > > > > a very high producer throughput, irrespective of the individual
> > > > messages
> > > > > > size.
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > 
> > > > > >
> > > > >
> > > >
>

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

[Enrico Olivelli]

Sijie,
> Sorry. When you say "we discussed", who are "we"? Is it DataStax?
There have been discussions on the ML and we discussed this also in
Community meetings.
So "we" is this community

> I believe we want to keep SQL until the code change lands in Trino.
Because
> there are still users using this component.

If we don't want to remove Presto then we have to at least move to the
latest version, that requires JDK11.

My understanding is that Lari is willing to help in completing this work.
When we discussed this, probably something like 1 year ago, Lari did not
have write permissions and he could not help.

There is not only Presto, we had to move all the Connectors and so change
the build process for the Pulsar "-all" docker image
We can defer Presto and move forward with the Connectors

Enrico



Il giorno gio 23 dic 2021 alle ore 06:34 Michael Marshall <
mmarsh...@apache.org> ha scritto:

> > I am volunteering to making the changes as per the PIP-62 plan.
>
> I think Lari is proposing to complete the work to move the
> Pulsar SQL code base to the repositories defined in PIP 62.
>
> Assuming that is what he meant, I am +1 on completing that now.
> Based on reading through the mailing list threads Lari referenced, we
> delayed moving the code base to the other repos with the hope that
> contributing the code to Trino/Presto would be done before the 2.8.0
> release. I think we should move forward with the initial PIP instead
> of waiting for the Trino PR to get merged.
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:39 PM Sijie Guo  wrote:
> >
> > Sorry. When you say "we discussed", who are "we"? Is it DataStax?
> >
> > I believe we want to keep SQL until the code change lands in Trino.
> Because
> > there are still users using this component.
> >
> > - Sijie
> >
> > On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli 
> > wrote:
> >
> > > Lari,
> > >
> > > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <
> lhot...@apache.org>
> > > ha scritto:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > > separate
> > > > repositories" was created in April 2020. The repositories for
> > > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about
> a
> > > year
> > > > ago [2].
> > > >
> > > > I'd like to propose that we continue with the PIP-62 plan asap,
> before
> > > > Apache Pulsar 2.10.0 is released.
> > > >
> > > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar
> without
> > > > waiting for the Trino Pulsar PR [3] being completed.
> > > >
> > > > I am volunteering to making the changes as per the PIP-62 plan.
> > > > When can we proceed with the plan? Do we need an explicit vote on the
> > > > mailing list about this?
> > > >
> > >
> > > We discussed this many times in the past year.
> > > I believe that there is no need to wait
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > BR,
> > > >
> > > > Lari
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > [2]
> > > >
> > > >
> > >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > > [3] https://github.com/trinodb/trino/pull/8020
> > > >
> > > > Other email threads:
> > > > * [Discuss] Don't include presto/trino in the normal Pulsar
> distribution
> > > -
> > > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > > * Updates on Presto connector for PIP-62 -
> > > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > > >
> > >
>

Re: [DISCUSS] Remove Grafana and Prometheus Dockerfiles; Remove DCOS example

[Enrico Olivelli]

+1

Enrico

Il Gio 23 Dic 2021, 19:01 Matteo Merli  ha scritto:

> +1
>
> On Thu, Dec 23, 2021 at 9:53 AM Michael Marshall 
> wrote:
>
> > Update: the PR [0] to remove the grafana docker image is now merged. I
> > still need reviews for the PR to remove the Prometheus docker image
> > [1]. I will follow up with a PR to remove the DCOS example.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/13389
> > [1] https://github.com/apache/pulsar/pull/13387
> >
> > On Mon, Dec 20, 2021 at 1:29 PM Michael Marshall 
> > wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > I would like to clean up our main project's `docker/` directory. We
> > > have a couple of old Dockerfiles that either need to be updated or
> > > removed. My vote is to remove them.
> > >
> > > 1. Grafana: with every release, we build and upload a custom grafana
> > > docker image just to package dashboards. Grafana provides other
> > > mechanisms to load dashboards. I propose we remove the grafana image
> > > from our release process and then move our dashboard json files to a
> > > new top level directory named `grafana/`. [0]
> > >
> > > Our image is based on grafana/grafana:4.3.1. The current grafana
> > > release is grafana/grafana:8.3.3.
> > >
> > > 2. Prometheus: we have a prometheus Dockerfile that was initially
> > > merged as part of a DCOS example 4 years ago. We have never published
> > > this docker image as part of a release [1]. There is nothing pulsar
> > > specific in these files. [2]
> > >
> > > Our image is based on prom/prometheus:v1.8.2. The latest prometheus
> > > image prom/prometheus:v2.32.1.
> > >
> > > 3. Can we remove our DCOS examples in `deployment/dcos`? We haven't
> > > updated our DCOS example deployment spec in 3.5 years. The current
> > > spec relies on our custom grafana docker image and on a community
> > > member's custom build [3] of our prometheus-dcos docker image. I don't
> > > think our example deployment should rely on a community member's
> > > custom docker image. (I don't have a PR to remove this example yet.)
> > >
> > > Alternatively, we could update the dockerfiles and the DCOS example.
> > > However, the lack of activity in keeping these three project
> > > dependencies up to date indicates to me that we should remove them.
> > >
> > > Let me know what you think.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/13389
> > > [1] https://hub.docker.com/u/apachepulsar/
> > > [2] https://github.com/apache/pulsar/pull/13387
> > > [3]
> >
> https://github.com/apache/pulsar/blob/c62ca808fc8c5aed3bb96b99bf6ef0c8ed382e3f/deployment/dcos/PulsarGroups.json#L251
> >
> --
> --
> Matteo Merli
> 
>

[ANNOUNCE] Apache Pulsar 2.9.1 released

[Enrico Olivelli]

The Apache Pulsar team is proud to announce Apache Pulsar version 2.9.1.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
http://pulsar.apache.org/release-notes

We would like to thank the contributors that made the release possible.

Regards,
The Pulsar Team

Re: [Vote] PIP 126: Support custom and pluggable consumer selector

[Enrico Olivelli]

zhangao,
I missed the discussion thread about this change.

did we discuss this somewhere ?
otherwise it is premature to start a VOTE

Enrico

Il giorno ven 24 dic 2021 alle ore 08:22 zhangao
 ha scritto:
>
> Hi Pulsar Community,   Please vote for this PIP.Best Regards,zhangao

[DISCUSS] PIP-124: Pulsar Client Shared State API

[Enrico Olivelli]

Hello everyone,
I want to start a discussion about PIP-124 Pulsar Client Shared  State API

This is the PIP document
https://github.com/apache/pulsar/issues/13490

This is a demo implementation (a proof-of-concept):
https://github.com/eolivelli/pulsar-shared-state-manager

Please take a look and share your thoughts

I believe that this will unlock the potential of the Exclusive
Producer and it will also make easier the life of many developers who
are using Pulsar and need some API to share configuration, metadata,
or any simple key-value data structure without adding a Database or
other components to their library, Pulsar IO connector or Pulsar
Protocol Handler.

Thanks
Enrico

Re: [DISCUSS] Apache Pulsar 2.10.0 release

[Enrico Olivelli]

PengHui,

Il giorno lun 27 dic 2021 alle ore 05:47 PengHui Li
 ha scritto:
>
> Hi, everyone
>
> I hope you’ve all been doing well. I would like to start an email thread to
> discuss features that we planned for 2.10.0.
> According to the time-based release plan
> https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan,
> we should release 2.10.0 at the end of December 2021, since we have reached
> the end of December,
> I would like to target the 2.10.0 to the end of January 2022

makes sense

Enrico

>
> There are some powerful features and enhancements in 2.10.0 such as
>
> - PIP 84: Message redelivery epoch
> - PIP 104: Add new consumer type: TableView
> - PIP 106: Negative acknowledgment backoff
> - PIP 110: Topic customized metadata support
> - PIP 117: Change Pulsar standalone defaults
> - PIP 118: Do not restart brokers when ZooKeeper session expires
> - PIP 119: Enable consistent hashing by default on KeyShared dispatcher
> - PIP 120: Enable client memory limit by default
> - PIP 121: Pulsar cluster level auto failover
> - PIP 123: Pulsar metadata CLI tool
> - Metadata service batch operations
> - RocksDB metadata service backend
> - Etcd metadata service backend
> - Ack timeout redelivery backoff policy
> - Global topic policies
>
> Most of them have been completed, some work in progress we need to try to
> complete within 2 weeks.
> This can give me a 2 week buffer period to prepare for release and complete
> the release vote.
> For the unfinished parts, we can move them to 2.11.0.
>
> Some proposals are just being discussed, so I do not list them because I'm
> not sure if we can complete them in two weeks.
>
> You can find all the change lists from
> https://github.com/apache/pulsar/pulls?q=milestone%3A2.10.0+-label%3Arelease%2F2.9.1
> There are more than 500 commits.
>
> If I missed something or you have any suggestions please let me know.
>
> Regards,
> Penghui

Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set

[Enrico Olivelli]

It looks loke you already prepared a awesome PIP!
Please copy your proposal on a PIP.

I am +1

Enrico

Il Mar 28 Dic 2021, 04:27 Zike Yang  ha
scritto:

> + 1 for skipping the topic level message size check for the chunk message
> in the broker.
>
> On Tue, Dec 28, 2021 at 9:14 AM guo jiwei  wrote:
>
> > +1
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> > On Tue, Dec 28, 2021 at 5:44 AM 陳智弘  wrote:
> > >
> > > +1
> > >
> > >  Skip the topic level setting is better
> > >
> > > Hang Chen  於 2021年12月27日 週一 22:09 寫道：
> > >
> > > > +1
> > > >
> > > > We'd better skip the topic level maxMessageSize check for chunk
> > messages.
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > PengHui Li  于2021年12月27日周一 22:07写道：
> > > > >
> > > > > +1,
> > > > >
> > > > > We can only skip the topic level messages size check for the chunk
> > > > message.
> > > > >
> > > > > Regards,
> > > > > Penghui
> > > > >
> > > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> > jianghait...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > I discovered a bug that chunk messages producing fails if topic
> > level
> > > > > > maxMessageSize is set [1]. The root cause of this issue is
> because
> > > > chunk
> > > > > > message is using broker level maxMessageSize as chunk size. And
> > topic
> > > > level
> > > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> > it is
> > > > set,
> > > > > > the on-going chunk message producing fails.
> > > > > >
> > > > > > ## Proposed changes
> > > > > > I would like to fix this by just skipping topic level
> > maxMessageSize
> > > > check
> > > > > > in
> > > > > >
> > > >
> >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > > level maxMessageSize is introduced in [2], for the purpose of
> > "easier
> > > > to
> > > > > > plan resource quotas for client allocation". And IMO this change
> > will
> > > > not
> > > > > > bring further complex into this.
> > > > > >
> > > > > > ## Alternative
> > > > > > Add a client side topic level maxMessageSize and keep it synced
> > with
> > > > > > broker.
> > > > > >
> > > > > > Required changes:
> > > > > > - [client] Add a new field
> > > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to
> store
> > this
> > > > > > client side topic level maxMessageSize.
> > > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > > CommandProducerSuccess for the initial value of
> > > > ProducerBase#maxMessageSize
> > > > > > - [PulsarApi.proto] Add a new Command like
> > > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> > updated.
> > > > > > Further more, some other data consistency issues need be handled
> > very
> > > > > > carefully when maxMessageSize is updated.
> > > > > > This alternative is complex but can also solve other topic level
> > > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> > case is
> > > > > > solved with PR [4]).
> > > > > >
> > > > > > Any suggestions or other use cases of topic level maxMessageSize
> > will
> > > > be
> > > > > > appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Haiting Jiang
> > > > > >
> > > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > > >
> > > >
> >
>
>
> --
> Zike Yang
>

Re: [DISCUSSION] PIP-130: Apply redelivery backoff policy for ack timeout

[Enrico Olivelli]

Penghui

I am overall +1 to this proposal but I am afraid about compatibility. Amd I
won't to be sure that we are not breaking anything.
Pulsar client and Consumer are configurable using a map of key value pairs.
So we must take care of not changing the behaviour.

What do you mean with 'redelivery backorder has not been released yet'?


Enrico

Il Lun 27 Dic 2021, 14:25 PengHui Li  ha scritto:

> https://github.com/apache/pulsar/issues/13528
>
> Pasted below for quoting convenience.
>
> -
>
> PIP 130: Apply redelivery backoff policy for ack timeout
>
> ## Motivation
>
> PIP 106
>
> https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff
> introduced negative acknowledgment message redelivery backoff which allows
> users to achieve
> more flexible message redelivery delay time control. But the redelivery
> backoff policy only
> apply to the negative acknowledgment API, for users who use ack timeout to
> trigger the message
> redelivery, not the negative acknowledgment API, they can't use the new
> features introduced by
> PIP 106.
>
> So the proposal is to apply the message redelivery policy for the ack
> timeout mechanism.
> Users can specify an ack timeout redelivery backoff, for example, apply an
> exponential backoff
> with 10 seconds ack timeout:
>
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
>
> The message redelivery behavior should be:
>
> |  Redelivery count   | Redelivery delay  |
> |    |   |
> | 1 | 10 + 1 seconds |
> | 2 | 10 + 2 seconds |
> | 3 | 10 + 4 seconds |
> | 4 | 10 + 8 seconds |
> | 5 | 10 + 16 seconds |
> | 6 | 10 + 32 seconds |
> | 7 | 10 + 60 seconds |
> | 8 | 10 + 60 seconds |
>
> ## Goal
>
> Add an API to the Java Client to provide the ability to specify the ack
> timeout message redelivery
> backoff and the message redelivery behavior should abide by the redelivery
> backoff policy.
>
>
> ## API Changes
>
> 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so that we
> can use the
> MessageRedeliveryBackoff for both negative acknowledgment API and ack
> timeout message redelivery.
>
> 2. Change `NegativeAckRedeliveryExponentialBackoff` to
> `ExponentialRedeliveryBackoff`, and add `multiplier`
> for `RedeliveryExponentialBackoff` with default value 2.
>
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6)
> .multiplier(5)
> .build()
>
> 3. Add `ackTimeoutRedeliveryBackoff` method for the `ConsumerBuilder`:
>
> ```java
> client.newConsumer()
> .ackTimeout(10, TimeUnit.SECOND)
> .ackTimeoutRedeliveryBackoff(
> ExponentialRedeliveryBackoff.builder()
> .minDelayMs(1000)
> .maxDelayMs(6).build());
> .subscribe();
> ```
>
> ## Compatibility and migration plan
>
> Since the negative acknowledgment message, redelivery backoff has not been
> released yet,
> so we can modify the API directly.
>
> ## Tests plan
>
> - Verify the introduced `multiplier` of ExponentialRedeliveryBackoff
> - Verify the ack timeout message redelivery work as expected
>
> Regards,
> Penghui
>

Re: [DISCUSSION] PIP-130: Apply redelivery backoff policy for ack timeout

[Enrico Olivelli]

+1

I missed that we haven't cherry picked that change to 2.9.

Thanks for your confirmation


Enrico

Il Mar 28 Dic 2021, 12:44 PengHui Li  ha scritto:

> Enrico
>
> > What do you mean with 'redelivery backorder has not been released yet'?
>
> I think you mean "redelivery backoff has not been released yet"?
>
> This proposal changed the public APIs, but the changed APIs are introduced
> by
> https://github.com/apache/pulsar/pull/12566, we don't have a release
> contains the
> APIs that introduced in #12566, so we are safe the API.
>
> The behavior will not be changed in this proposal, we just have an
> additional API for users
> to achieve the flexible message redelivery control, will not change the
> behavior we have.
>
> > Pulsar client and Consumer are configurable using a map of key value
> pairs.
> So we must take care of not changing the behaviour.
>
> The proposal will not add more fields to the consumer configuration data,
> Just change the class name here
>
> https://github.com/apache/pulsar/blob/f0da648ae1c02248c015d26b93e08b2a9a78c1d3/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java#L78
>
> Regards,
> Penghui
>
>
> On Tue, Dec 28, 2021 at 3:19 PM Enrico Olivelli 
> wrote:
>
> > Penghui
> >
> > I am overall +1 to this proposal but I am afraid about compatibility.
> Amd I
> > won't to be sure that we are not breaking anything.
> > Pulsar client and Consumer are configurable using a map of key value
> pairs.
> > So we must take care of not changing the behaviour.
> >
> > What do you mean with 'redelivery backorder has not been released yet'?
> >
> >
> > Enrico
> >
> > Il Lun 27 Dic 2021, 14:25 PengHui Li  ha scritto:
> >
> > > https://github.com/apache/pulsar/issues/13528
> > >
> > > Pasted below for quoting convenience.
> > >
> > > -
> > >
> > > PIP 130: Apply redelivery backoff policy for ack timeout
> > >
> > > ## Motivation
> > >
> > > PIP 106
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-106%3A-Negative-acknowledgment-backoff
> > > introduced negative acknowledgment message redelivery backoff which
> > allows
> > > users to achieve
> > > more flexible message redelivery delay time control. But the redelivery
> > > backoff policy only
> > > apply to the negative acknowledgment API, for users who use ack timeout
> > to
> > > trigger the message
> > > redelivery, not the negative acknowledgment API, they can't use the new
> > > features introduced by
> > > PIP 106.
> > >
> > > So the proposal is to apply the message redelivery policy for the ack
> > > timeout mechanism.
> > > Users can specify an ack timeout redelivery backoff, for example, apply
> > an
> > > exponential backoff
> > > with 10 seconds ack timeout:
> > >
> > > ```java
> > > client.newConsumer()
> > > .ackTimeout(10, TimeUnit.SECOND)
> > > .ackTimeoutRedeliveryBackoff(
> > > ExponentialRedeliveryBackoff.builder()
> > > .minDelayMs(1000)
> > > .maxDelayMs(6).build());
> > > .subscribe();
> > > ```
> > >
> > > The message redelivery behavior should be:
> > >
> > > |  Redelivery count   | Redelivery delay  |
> > > |    |   |
> > > | 1 | 10 + 1 seconds |
> > > | 2 | 10 + 2 seconds |
> > > | 3 | 10 + 4 seconds |
> > > | 4 | 10 + 8 seconds |
> > > | 5 | 10 + 16 seconds |
> > > | 6 | 10 + 32 seconds |
> > > | 7 | 10 + 60 seconds |
> > > | 8 | 10 + 60 seconds |
> > >
> > > ## Goal
> > >
> > > Add an API to the Java Client to provide the ability to specify the ack
> > > timeout message redelivery
> > > backoff and the message redelivery behavior should abide by the
> > redelivery
> > > backoff policy.
> > >
> > >
> > > ## API Changes
> > >
> > > 1. Change `NegativeAckRedeliveryBackoff` to `RedeliveryBackoff`, so
> that
> > we
> > > can use the
> > > MessageRedeliveryBackoff for both negative acknowledgment API and ack
> > > timeout message redelivery.
> > >
> > > 2. Change `NegativeAckRedeliveryExponentialBackoff` to
> > > `ExponentialRedeliveryBackoff`, and add `multiplier`
> > > for `RedeliveryExponentialBackoff` w

Re: [DISCUSS] PIP-124: Pulsar Client Shared State API

[Enrico Olivelli]

Matteo,

Il Mer 29 Dic 2021, 02:57 Matteo Merli  ha scritto:

> > * Add an API to the Java client that makes it easier to maintain a
> consistent Share State between instances of an application.
> > * Provide some ready to use recipes, like a simple key-value store
> >
> > It is not a goal to implement a Pulsar backed Database system
>
> While the first use case for Pulsar was indeed to be the
> messaging/replication platform for a distributed database, and it has
> been working in production for many years, I'm not convinced to add
> this level of API as part of the Pulsar client API.
>
> Pulsar API has been designed to be high-level and easy to use (and
> reason about), with in mind the use cases of application developers. I
> don't think that a "storage" level API fits well with the rest of
> abstractions.
>
> > public interface PulsarMap extends AutoCloseable {
> > ..
> >  CompletableFuture put(K key, V value)
>
> If all the logic is implemented in the client side, when there are
> multiple clients sharing the same, how can any of them mutate the
> state, since we actually enforce that there is a single exclusive
> producer? Would a user get an error if there's already a different
> client writing?
>
> My impression is that, while looking convenient, a shared Map
> interface is not the best abstraction for either case:
>  * If you're actually building a DB, you will definitely need access
> to the log itself rather than a Map interface
>  * If you want to share some state across multiple processes without
> using a DB, there are many tricky API, consistency and semantic
> problems to solve, many of which are just pushed down to the
> application which will need to be aware and understand them. At that
> point, I would seriously recommend using a DB, or if the question is:
> "I don't want to use an additional external system", then to use the
> BK TableService component.
>

This is usually not a option because the BK TableService does not support
well multi tenancy and also the application would need to connect to the
Bookies (think about configuration, security...)


>
> I think this feature should be best viewed as a recipe, as it doesn't
> depend on or benefits from any internal broker support. If there are
> enough interest and concrete use cases it can be then included later.
>

My initial proposal was to push this to Pulsar Adapters.
I changed the proposal before sending the PIP because I think it very
useful for Protocol Handlers  and in Pulsar IO connectors.

I am totally fine to add this to pulsar-adapters, but I want to see this in
the Pulsar repo and released as part of an official Pulsar recipe.

@Matteo does this sound like a good option to you?

Otherwise we miss the possibility to make it easier for Pulsar users to
leverage this power of Pulsar.

In Pravega you have State Synchronizers and they are a great foundational
API and we are missing something like that in Pulsar.

Enrico



> --
> Matteo Merli
> 
>
> On Fri, Dec 24, 2021 at 1:53 AM Enrico Olivelli 
> wrote:
> >
> > Hello everyone,
> > I want to start a discussion about PIP-124 Pulsar Client Shared  State
> API
> >
> > This is the PIP document
> > https://github.com/apache/pulsar/issues/13490
> >
> > This is a demo implementation (a proof-of-concept):
> > https://github.com/eolivelli/pulsar-shared-state-manager
> >
> > Please take a look and share your thoughts
> >
> > I believe that this will unlock the potential of the Exclusive
> > Producer and it will also make easier the life of many developers who
> > are using Pulsar and need some API to share configuration, metadata,
> > or any simple key-value data structure without adding a Database or
> > other components to their library, Pulsar IO connector or Pulsar
> > Protocol Handler.
> >
> > Thanks
> > Enrico
>

Re: [VOTE] Apache Pulsar 2.8.2 candidate 2

[Enrico Olivelli]

What's the status of this VOTE?

Enrico

Il Mer 22 Dic 2021, 10:34 Nicolò Boschi  ha scritto:

> +1 (non binding)
>
> Checks:
> - Checksum and signatures
> - Apache Rat check passes
> - Compile from source w JDK8
> - Build docker image from source
> - Run Pulsar standalone and produce-consume from CLI
> - Verified Log4J inside lib/
>
> -rw-r--r-- 1 root root   208235 Jan 22  2020
> org.apache.logging.log4j-log4j-1.2-api-2.17.0.jar
>
> -rw-r--r-- 1 root root   301776 Jan 22  2020
> org.apache.logging.log4j-log4j-api-2.17.0.jar
>
> -rw-r--r-- 1 root root  1789339 Jan 22  2020
> org.apache.logging.log4j-log4j-core-2.17.0.jar
>
> -rw-r--r-- 1 root root24252 Jan 22  2020
> org.apache.logging.log4j-log4j-slf4j-impl-2.17.0.jar
>
> -rw-r--r-- 1 root root35920 Jan 22  2020
> org.apache.logging.log4j-log4j-web-2.17.0.jar
>
> Il giorno mer 22 dic 2021 alle ore 06:37 Lin Lin  ha
> scritto:
>
> >
> >
> > On 2021/12/21 10:48:41 Shivji Kumar Jha wrote:
> > > Hi LinLin,
> > >
> > > Log4j version 2.16.0 has DDoS possibilities in some cases [1] . Can we
> > move
> > > to Log4j 2.17.0 in 2.8.2?
> > >
> > > Apache Log4j2 versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3) did
> > not
> > > > protect from uncontrolled recursion from self-referential lookups.
> This
> > > > allows an attacker with control over Thread Context Map data to
> cause a
> > > > denial of service when a crafted string is interpreted. This issue
> was
> > > > fixed in Log4j 2.17.0 and 2.12.3.
> >
> >
> > Already included
> >
>
>
> --
> Nicolò Boschi
>

Sending 2.6 release line to EOF

[Enrico Olivelli]

Hello,
We are no more cutting releases out of branch-2.6

We must declare 2.6 EOF.

IIUC there is no process established for this at the moment in the Pulsar
project.

So for this time I will call out a VOTE next week, as we did not cut
releases for a few important bugs.

Thoughts?

Enrico

Re: [VOTE] PIP-130: Apply redelivery backoff policy for ack timeout

[Enrico Olivelli]

+1 (binding)

Enrico

Il Mar 4 Gen 2022, 16:19 刘德志  ha scritto:

>
>
> +1
>
> Thanks,
> Dezhi Liu

Re: [DISCUSS] Apache Pulsar 2.9.2 release

[Enrico Olivelli]

171 commits?

Why?

This is too much in my opinion for a point release.

I am pretty sure that we don't need all that changes.

Every change we cherry pick has a good chance to break the stability

Enrico

Il Mer 5 Gen 2022, 16:30 mattison chao  ha scritto:

> +1
>
> regards,
> Mattisonchao
>
> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
>
> > +1,Thanks for your work.
> >
> >
> >
> >
> >
> >
> > Best Regards,
> > Lan Liang
> > On 1/5/2022 23:01，Hang Chen wrote：
> > +1
> >
> > Best,
> > Hang
> >
> > 陳智弘  于2022年1月5日周三 21:30写道：
> >
> > +1
> >
> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
> >
> > +1
> >
> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
> >
> > Hello, Pulsar community:
> >
> > I'd like to propose that we release Apache Pulsar 2.9.2.
> >
> > Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer
> > to [0]), they contain the log4j security patch and many important fixes.
> >
> > I am happy to volunteer to be the release manager.
> >
> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
> > labeled release/2.9.2, I'll follow them to make sure important fix could
> > be
> > merged in 2.9.2.
> >
> >
> > Best,
> > Ran Gao
> >
> >
> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
> >
> >
> >
>

Re: [DISCUSS] Apache Pulsar 2.9.2 release

[Enrico Olivelli]

BTW +1

I hope we will review carefully what we are including in the release

And that we will let branch-2.9 stabilise by not adding any other commits
that are not strictly required:
- security related issues
- data loss/data corruption cases


Enrico


Thank you Ran Gao for starting this thread.



Enrico

Il Mer 5 Gen 2022, 17:29 Enrico Olivelli  ha scritto:

> 171 commits?
>
> Why?
>
> This is too much in my opinion for a point release.
>
> I am pretty sure that we don't need all that changes.
>
> Every change we cherry pick has a good chance to break the stability
>
> Enrico
>
> Il Mer 5 Gen 2022, 16:30 mattison chao  ha
> scritto:
>
>> +1
>>
>> regards,
>> Mattisonchao
>>
>> On Wed, 5 Jan 2022 at 23:04, Lan Liang  wrote:
>>
>> > +1,Thanks for your work.
>> >
>> >
>> >
>> >
>> >
>> >
>> > Best Regards,
>> > Lan Liang
>> > On 1/5/2022 23:01，Hang Chen wrote：
>> > +1
>> >
>> > Best,
>> > Hang
>> >
>> > 陳智弘  于2022年1月5日周三 21:30写道：
>> >
>> > +1
>> >
>> > Lari Hotari  於 2022年1月5日 週三 20:56 寫道：
>> >
>> > +1
>> >
>> > On Wed, Jan 5, 2022 at 11:23 AM Ran Gao  wrote:
>> >
>> > Hello, Pulsar community:
>> >
>> > I'd like to propose that we release Apache Pulsar 2.9.2.
>> >
>> > Currently, compared to 2.9.1, branch-2.9 already merged 171
>> commits(refer
>> > to [0]), they contain the log4j security patch and many important fixes.
>> >
>> > I am happy to volunteer to be the release manager.
>> >
>> > I see 4 merged PRs are labeled release/2.9.2 but not cherry-pick to
>> > branch-2.9, I'll cherry-pick them, and there are also 20 open PRs are
>> > labeled release/2.9.2, I'll follow them to make sure important fix could
>> > be
>> > merged in 2.9.2.
>> >
>> >
>> > Best,
>> > Ran Gao
>> >
>> >
>> > [0] https://github.com/apache/pulsar/compare/v2.9.1...branch-2.9
>> >
>> >
>> >
>>
>

Re: [DISCUSSION] PIP-121: Pulsar cluster level auto failover

[Enrico Olivelli]

I am commeting on the GH issue

Thanks

Enrico

Il Mer 5 Gen 2022, 04:56 PengHui Li  ha scritto:

> +1
>
> Penghui
>
> On Tue, Jan 4, 2022 at 4:51 PM Hang Chen  wrote:
>
> > https://github.com/apache/pulsar/issues/13315
> >
> > Pasted below for quoting convenience.
> >
> > 
> > ### Motivation
> > We have geo-replication to support Pulsar cluster level failover. We
> > can setup Pulsar cluster A as a primary cluster in data center A, and
> > setup Pulsar cluster B as backup cluster in data center B. Then we
> > configure geo-replication between cluster A and cluster B. All the
> > clients are connected to the Pulsar cluster by DNS. If cluster A is
> > down, we should switch the DNS to point the target Pulsar cluster from
> > cluster A to cluster B. After the clients are resolved to cluster B,
> > they can produce and consume messages normally. After cluster A
> > recovers, the administrator should switch the DNS back to cluster A.
> >
> > However, the current method has two shortcomings.
> > 1. The administrator should monitor the status of all Pulsar clusters,
> > and switch the DNS as soon as possible when cluster A is down. The
> > switch and recovery is not automatic and recovery time is controlled
> > by the administrator, which will put the administrator under heavy
> > load.
> > 2. The Pulsar client and DNS system have a cache. When the
> > administrator switches the DNS from cluster A to Cluster B, it will
> > take some time for cache trigger timeout, which will delay client
> > recovery time and lead to the product/consumer message failing.
> >
> > ### Goal
> > It's better to provide an automatic cluster level failure recovery
> > mechanism to make pulsar cluster failover more effective. We should
> > support pulsar clients auto switching from cluster A to cluster B when
> > it detects cluster A has been down according to the configured
> > detecting policy and switch back to cluster A when it has recovered.
> > The reason why we should switch back to cluster A is that most
> > applications may be deployed in data center A and they have low
> > network cost for communicating with pulsar cluster A. If they keep
> > visiting pulsar cluster B, they have high network cost, and cause high
> > produce/consume latency.
> >
> > In order to improve the DNS cache problem, we should provide an
> > administrator controlled switch provider for administrators to update
> > service URLs.
> >
> > In the end, we should provide an auto service URL switch provider and
> > administrator controlled switch provider.
> >
> > ### Design
> > We have already provided the `ServiceUrlProvider` interface to support
> > different service URLs. In order to support automatic cluster level
> > failure auto recovery, we can provide different ServiceUrlProvider
> > implementations. For current requirements, we can provide
> > `AutoClusterFailover` and `ControlledClusterFailover`.
> >
> >  AutoClusterFailover
> > In order to support auto switching from the primary cluster to the
> > secondary, we can provide a probe task, which will probe the activity
> > of the primary cluster and the secondary one. When it finds the
> > primary cluster failed more than `failoverDelayMs`, it will switch to
> > the secondary cluster by calling `updateServiceUrl`. After switching
> > to the secondary cluster, the `AutoClusterFailover` will continue to
> > probe the primary cluster. If the primary cluster comes back and
> > remains active for `switchBackDelayMs`, it will switch back to the
> > primary cluster.
> > The APIs are listed as follows.
> >
> > In order to support multiple secondary clusters, use List to store
> > secondary cluster urls. When the primary cluster probe fails for
> > failoverDelayMs, it will start to probe the secondary cluster list one
> > by one, once it finds the active cluster, it will switch to the target
> > cluster. Notice: If you configured multiple clusters, you should turn
> > on cluster level geo-replication to ensure the topic data sync between
> > all primary and secondary clusters. Otherwise, it may distribute the
> > topic data into different clusters. And the consumers won’t get the
> > whole data of the topic.
> >
> > In order to support different authentication configurations between
> > clusters, we provide the authentication relation configurations
> > updated with the target cluster.
> >
> > ```Java
> > public class AutoClusterFailover implements ServiceUrlProvider {
> >
> >private AutoClusterFailover(String primary, List secondary,
> > long failoverDelayNs, long switchBackDelayNs,
> > long intervalMs, Authentication
> > primaryAuthentication,
> > List
> > secondaryAuthentications, String primaryTlsTrustCertsFilePath,
> > List
> > secondaryTlsTrustCertsFilePaths, String primaryTlsTrustStorePath,
> > List
> > secondaryTlsTrustStorePaths, String primaryTlsTrustStorePasswor

Re: About the implementation of a Mail Queue using Pulsar in Apache James

[Enrico Olivelli]

Benoit,

Il Gio 6 Gen 2022, 10:54 Benoit TELLIER  ha scritto:

> Hello Pulsar folks!
>
> I am a member of the Apache James PMC, whose maintain the Java Apache
> Mail Extensible Server (for the intimates hence James). The Distributed
> James server aims at scaling emails on top of modern databases and
> messaging technologies.
>
> So far we relies on ActiveMQ/RabbitMQ as a messaging technology (please
> don't judge us...). Features includes:
>  - A mail queue (~work queue) supporting browsing, removes, delays
>  - A messaging system for notifications (think IMAP IDLE spec)
>  - Some smaller stuff: work-queues, ...
>
> However this matter of fact will hopefully change as we receive a
> contribution for a Mail Queue implemented with Pulsar [1].
>


This is great.

I have been working on mail queue systems for years and I will be happy to
review the PR and help with this opportunity to engage with the James
community

Cheers
Enrico


> As an Apache enthusiast, I think this could be an opportunity to
> strengthen links between our communities. Who knows, some devs within
> the Pulsar community might feel interested by this, and could share some
> interesting insights on this. The Apache James PMC is mostly
> Pulsar-naive so we might also benefit from an external eyes. In the
> future we might also propose some work (eg GSOC 2022) on further
> integrationg Pulsar into James, which could be an opportunity for
> cooperation.
>
> I apologize if this mail is of no interest to the Apache Pulsar
> community or if it is the wrong place to post this kind of stuff.
>
> Best regards,
>
> Benoit TELLIER
>
> [1] The aforementioned Pulsar PR :
> https://github.com/apache/james-project/pull/808
>
>

Re: [ANNOUNCE] Apache Pulsar 2.8.2 released

[Enrico Olivelli]

Thanks!

Congrats

Enrico

Il Gio 6 Gen 2022, 13:26 PengHui Li  ha scritto:

> Thanks for the great work!
>
> Regards,
> Penghui
>
> On Wed, Jan 5, 2022 at 10:16 PM linlin  wrote:
>
> > The Apache Pulsar team is proud to announce Apache Pulsar version 2.8.2.
> >
> > Pulsar is a highly scalable, low latency messaging platform running on
> > commodity hardware. It provides simple pub-sub semantics over topics,
> > guaranteed at-least-once delivery of messages, automatic cursor
> management
> > for
> > subscribers, and cross-datacenter replication.
> >
> > For Pulsar release details and downloads, visit:
> >
> > https://pulsar.apache.org/download
> >
> > Release Notes are at:
> > http://pulsar.apache.org/release-notes
> >
> > We would like to thank the contributors that made the release possible.
> >
> > Regards,
> >
> > The Pulsar Team
> >
>

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

[Enrico Olivelli]

Sijie,

Il Ven 7 Gen 2022, 01:58 Sijie Guo  ha scritto:

> The fundamental problem I see is that we don't have a proper helm chart
> release and we don't have good versioning guidance.
>
> Chart should have its own version, which is independent of the Pulsar
> version.
>
> Also, we need to provide a guide to the community - most of the time, you
> don't need to upgrade zookeeper and bookkeeper. Because these two
> components are rarely changed.
>

I agree with you and I want to add that one of the problems is that we are
using the Pulsar docker images to start Zookeeper and Bookies.
And we are wrapping the entrypoints (I mean the command line to launch the
service) with the Pulsar command.
So the version you use for zookeeper and BookKeeper is tied to the Pulsar
version.
Usually it is hard to explain to users to use Pulsar 2.x for the bookies
and Pulsar 2.y for the brokers.

We should move to using the standard docker images for zookeeper and
BookKeeper.




> That means:
>
> 1. We should have a separate `version` from `appVersion`.
> 2. We should use the Pulsar image version as the `appVersion`.
> 3. It is okay to only update broker and proxy images version and leave
> zookeeper and bookkeeper version unchanged.
>
> That's probably not the final guide. But at least, we can have something to
> get started to formalize the process. This guide will then help us handle
> such situations better.
>

I would fix the helm chart now and t let people upgrade to Pulsar 2.8/2.9
and then we can provide a better solution.

Otherwise people are forced to use third party helm charts, and I really
don't like this situation

Enrico



> - Sijie
>
> On Wed, Jan 5, 2022 at 6:00 AM 陳智弘  wrote:
>
> > Hi everyone,
> >
> >   From my side, I think currently merging the request without regarding
> the
> > known issue and announcing this information on the website is a good
> > option.
> >
> > <
> >
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> > >
> > 不含病毒。www.avg.com
> > <
> >
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> > >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
> > Lari Hotari  於 2022年1月5日 週三 下午9:04寫道：
> >
> > > Reminder: we need to decide about the developement of Apache Pulsar
> Helm
> > > chart.
> > > Please reply to the email or review
> > > https://github.com/apache/pulsar-helm-chart/pull/190 .
> > > That PR is blocked since a decision must be made whether it's fine to
> > make
> > > the change, although there's a known issue in Zookeeper when TLS is
> > > enabled.
> > > The issue is ZOOKEEPER-3988 /
> > > https://github.com/apache/pulsar/issues/11070 .
> > > The bug currently only impacts TLS since the change
> > > https://github.com/apache/pulsar-helm-chart/pull/180 switched
> > > to use NIOServerCnxnFactory for Zookeeper. NIOServerCnxnFactory doesn't
> > > support TLS and the impacted NettyServerCnxnFactory must be used for
> TLS.
> > >
> > > How do we handle the decision? Can we proceed in merging
> > > https://github.com/apache/pulsar-helm-chart/pull/190 regardless of the
> > > known issue?
> > >
> > > BR,
> > > Lari
> > >
> > > On Mon, Jan 3, 2022 at 4:39 PM Lari Hotari  wrote:
> > >
> > > > Hi all,
> > > >
> > > > There's an upcoming change in the Apache Pulsar Helm chart to finally
> > > > switch to Pulsar 2.8.x, more specifically to Apache Pulsar version
> > 2.8.2
> > > .
> > > > The latest Apache Pulsar Helm Chart release uses the Apache Pulsar
> > 2.7.4
> > > > image.
> > > >
> > > > The pull request to switch to Apache Pulsar image version 2.8.2 is
> > > current
> > > > in review:
> > > > https://github.com/apache/pulsar-helm-chart/pull/190
> > > >
> > > > There's a known issue that Zookeeper TLS isn't stable because of
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-3988 (also reported
> in
> > > > apache/pulsar as https://github.com/apache/pulsar/issues/11070) .
> > > > The fix https://github.com/apache/zookeeper/pull/1770 is planned for
> > > > Zookeeper 3.7.1 version.
> > > > There's a workaround in the Apache Pulsar Helm chart when TLS isn't
> > > > enabled for Zookeeper. That was added by
> > > > https://github.com/apache/pulsar-helm-chart/pull/180 .
> > > > However, the workaround cannot be applied when TLS is enabled for
> > > > Zookeeper.
> > > >
> > > > Should we postpone switching to Apache Pulsar 2.8.2 in the Helm chart
> > > > until there's a fix for ZOOKEEPER-3988 /
> > > > https://github.com/apache/pulsar/issues/11070 ?
> > > >
> > > > BR,
> > > > Lari
> > > >
> > >
> >
>

[LAZY CONSENSUS] Sending 2.6 release line to EOF

[Enrico Olivelli]

This thread is now to moving branch-2.6 (the Pulsar 2.6 release line)
to End-Of-Life status

What EOL means here:
- No more releases, even for security bugs
- No cherry-picks to branch-2.6

We will apply the Lazy Consensus rules
https://www.apache.org/foundation/glossary.html#LazyConsensus

If no-one objects within 72 hours I will send an official announcement
of the EOL status of 2.6.

Enrico


Il giorno mar 4 gen 2022 alle ore 06:04 Michael Marshall
 ha scritto:
>
> I agree with Dave. Lazy consensus gives us exactly what we need: to
> know if someone would like branch-2.6 to stay alive. If no one
> responds that they need it within the time period, then no one needs
> it.
>
> PIP 47 briefly mentions our EOL policy, but the policy doesn't cover
> our current case where we released fewer than 4 minor versions in a
> year. I think we'd benefit from clarifying and improving our
> documentation for our EOL policy.
>
> > This project VOTEs on more things than any Apache project I’ve been 
> > involved with during the last 14 years.
>
> The new PIP process, with its increased purview, has increased the
> number of votes. I have appreciated the increased development
> discussion on the ML, but maybe the discussions (especially the ones
> that look like votes) and then the subsequent votes are leading to the
> high vote volume.
>
> We could update the PIP process so that DISCUSSIONS without any
> pushback during a certain time period do not need VOTES.
>
> Thanks,
> Michael
>
>
> On Mon, Jan 3, 2022 at 6:00 PM Dave Fisher  wrote:
> >
> > This project VOTEs on more things than any Apache project I’ve been 
> > involved with during the last 14 years.
> >
> > For most projects lazy consensus applies: 
> > https://www.apache.org/foundation/glossary.html#LazyConsensus
> >
> > We also tend to “vote” during discussions which does not really advance a 
> > conversation.
> >
> > If we want to have a VOTE then there are good reasons to start it NOW.
> >
> > All the Best,
> > Dave
> >
> > > On Jan 3, 2022, at 3:48 PM, Sijie Guo  wrote:
> > >
> > > Agree to have a vote to keep a record.
> > >
> > > - Sijie
> > >
> > > On Mon, Jan 3, 2022 at 3:40 PM 陳智弘  wrote:
> > >
> > >> I think having a vote and quickly announce the EOF of 2.6.x will be 
> > >> better
> > >> to the community.
> > >>
> > >>
> > >> Dave Fisher  於 2022年1月4日 週二 06:06 寫道：
> > >>
> > >>> I don’t think we need a VOTE. Let’s do this by LAZY CONSENSUS.
> > >>>
> > >>>> On Jan 3, 2022, at 10:05 AM, Enrico Olivelli 
> > >>> wrote:
> > >>>>
> > >>>> Hello,
> > >>>> We are no more cutting releases out of branch-2.6
> > >>>>
> > >>>> We must declare 2.6 EOF.
> > >>>>
> > >>>> IIUC there is no process established for this at the moment in the
> > >> Pulsar
> > >>>> project.
> > >>>>
> > >>>> So for this time I will call out a VOTE next week, as we did not cut
> > >>>> releases for a few important bugs.
> > >>>>
> > >>>> Thoughts?
> > >>>>
> > >>>> Enrico
> > >>>
> > >>>
> > >>
> >

Re: [VOTE] PIP-121: Pulsar cluster level auto failover on client side

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno lun 10 gen 2022 alle ore 07:45 Hang Chen
 ha scritto:
>
> This is the voting thread for PIP-121. It will stay open for at least 48
> hours.
>
> https://github.com/apache/pulsar/issues/13315
>
> Pasted below for quoting convenience.
>
> -
> ### Motivation
> We have geo-replication to support Pulsar cluster level failover. We
> can set up Pulsar cluster A as a primary cluster in data center A, and
> setup Pulsar cluster B as backup cluster in data center B. Then we
> configure geo-replication between cluster A and cluster B. All the
> clients are connected to the Pulsar cluster by DNS. If cluster A is
> down, we should switch the DNS to point the target Pulsar cluster from
> cluster A to cluster B. After the clients are resolved to cluster B,
> they can produce and consume messages normally. After cluster A
> recovers, the administrator should switch the DNS back to cluster A.
>
> However, the current method has two shortcomings.
> 1. The administrator should monitor the status of all Pulsar clusters,
> and switch the DNS as soon as possible when cluster A is down. The
> switch and recovery is not automatic and recovery time is controlled
> by the administrator, which will put the administrator under heavy
> load.
> 2. The Pulsar client and DNS system have a cache. When the
> administrator switches the DNS from cluster A to Cluster B, it will
> take some time for cache trigger timeout, which will delay client
> recovery time and lead to the product/consumer message failing.
>
> ### Goal
> It's better to provide an automatic cluster level failure recovery
> mechanism to make pulsar cluster failover more effective. We should
> support pulsar clients auto switching from cluster A to cluster B when
> it detects cluster A has been down according to the configured
> detecting policy and switch back to cluster A when it has recovered.
> The reason why we should switch back to cluster A is that most
> applications may be deployed in data center A and they have low
> network cost for communicating with pulsar cluster A. If they keep
> visiting pulsar cluster B, they have high network cost, and cause high
> produce/consume latency.
>
> In order to improve the DNS cache problem, we should provide an
> administrator controlled switch provider for administrators to update
> service URLs.
>
> In the end, we should provide an auto service URL switch provider and
> administrator controlled switch provider.
>
> ### Design
> We have already provided the `ServiceUrlProvider` interface to support
> different service URLs. In order to support automatic cluster level
> failure auto recovery, we can provide different ServiceUrlProvider
> implementations. For current requirements, we can provide
> `AutoClusterFailover` and `ControlledClusterFailover`.
>
>  AutoClusterFailover
> In order to support auto switching from the primary cluster to the
> secondary, we can provide a probe task, which will probe the activity
> of the primary cluster and the secondary one. When it finds the
> primary cluster failed more than `failoverDelayMs`, it will switch to
> the secondary cluster by calling `updateServiceUrl`. After switching
> to the secondary cluster, the `AutoClusterFailover` will continue to
> probe the primary cluster. If the primary cluster comes back and
> remains active for `switchBackDelayMs`, it will switch back to the
> primary cluster.
> The APIs are listed as follows.
>
> In order to support multiple secondary clusters, use List to store
> secondary cluster urls. When the primary cluster probe fails for
> failoverDelayMs, it will start to probe the secondary cluster list one
> by one, once it finds the active cluster, it will switch to the target
> cluster. Notice: If you configured multiple clusters, you should turn
> on cluster level geo-replication to ensure the topic data sync between
> all primary and secondary clusters. Otherwise, it may distribute the
> topic data into different clusters. And the consumers won’t get the
> whole data of the topic.
>
> In order to support different authentication configurations between
> clusters, we provide the authentication relation configurations
> updated with the target cluster.
>
> ```Java
> public class AutoClusterFailover implements ServiceUrlProvider {
>
>private AutoClusterFailover(AutoClusterFailoverBuilderImpl builder) {
> //
> }
>
> @Override
> public void initialize(PulsarClient client) {
> this.pulsarClient = client;
>
> // start to probe primary cluster active or not
> executor.scheduleAtFixedRate(catchingAndLoggingThrowables(() -> {
> // probe and switch
> }), intervalMs, intervalMs, TimeUnit.MILLISECONDS);
>
> }
>
> @Override
> public String getServiceUrl() {
> return this.currentPulsarServiceUrl;
> }
>
> @Override
> public void close() {
> this.executor.shutdown();
> }
>
> // probe pulsar cluster available
> private boolea

Re: [VOTE] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno lun 10 gen 2022 alle ore 07:47 Hang Chen
 ha scritto:

>
> This is the voting thread for PIP-122. It will stay open for at least 48
> hours.
>
> https://github.com/apache/pulsar/issues/13340
>
> Pasted below for quoting convenience.
>
> 
>
> ### Motivation
> The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> https://github.com/apache/pulsar/pull/6772. It can resolve many load
> balance issues of `OverloadShedder` and works well in many Pulsar
> production clusters.
>
> In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> policy is `OverloadShedder`.
>
> I think it's a good time for 2.10 to change default load balance
> policy to `ThresholdShedder`, it will make throughput more balance
> between brokers.
>
> ### Proposed Changes
> In 2.10 release，for `broker.conf`, change
> `loadBalancerLoadSheddingStrategy` from
> `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`

Re: [DISCUSSION] PIP-133 Pulsar Functions Add API For Accessing Other Function States

[Enrico Olivelli]

Thank you for posting your PIP !

I am sharing some of Neng's concerns.
We should define clearly how security works.

Also, currently the function defines some "namespace" for the state
storage, and we recently added support for custom state storage
implementation. With this change each function will possibly access
other state storage namespaces (think about using a Database per each
tenant).

We should also state something about guarantees while accessing
multiple storages and/or about transactional (atomic?) access


Enrico

Il giorno mar 11 gen 2022 alle ore 21:38 Neng Lu  ha scritto:
>
> Before we advance further, we first need to get on the same page of the
> pros and cons of allowing this feature.
>
> If functions can access (especially the write access) other functions'
> state, the data ownership will be a mess, isolation is broken and data
> security might be compromised.
>
>
>
>
>
> On Wed, Jan 5, 2022 at 3:45 PM Ethan Merrill 
> wrote:
>
> > Original PIP: https://github.com/apache/pulsar/issues/13633
> >
> > Pasted below for quoting convenience.
> >
> > -
> >
> > ## Motivation
> >
> > Certain uses of Pulsar functions could benefit from the ability to access
> > the states of other functions. Currently functions can only access their
> > own states, and so sharing information between functions requires other
> > solutions such as writing to a separate database.
> >
> > ## Goal
> >
> > The goal is to enable the ability for a function to access another
> > function's state. Given another function's tenant, namespace, and name, any
> > function should be able to access the other function's state for read and
> > write purposes. This PIP is not concerned with expanding the capabilities
> > of function states, It only deals with expanding access to function states.
> >
> > ## API Changes
> >
> > The Pulsar function API would be modified to have the function context
> > implement the following interface for accessing function states using a
> > tenant, namespace, and name.
> >
> > ```
> > public interface SharedContext {
> > /**
> >  * Update the state value for the key.
> >  *
> >  * @param key   name of the key
> >  * @param value state value of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  */
> > void putState(String key, ByteBuffer value, String tenant, String ns,
> > String name);
> >
> > /**
> >  * Update the state value for the key, but don't wait for the
> > operation to be completed
> >  *
> >  * @param key   name of the key
> >  * @param value state value of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  */
> > CompletableFuture putStateAsync(String key, ByteBuffer value,
> > String tenant, String ns, String name);
> >
> > /**
> >  * Retrieve the state value for the key.
> >  *
> >  * @param key name of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  * @return the state value for the key.
> >  */
> > ByteBuffer getState(String key, String tenant, String ns, String name);
> >
> > /**
> >  * Retrieve the state value for the key, but don't wait for the
> > operation to be completed
> >  *
> >  * @param key name of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  * @return the state value for the key.
> >  */
> > CompletableFuture getStateAsync(String key, String tenant,
> > String ns, String name);
> >
> > /**
> >  * Delete the state value for the key.
> >  *
> >  * @param key   name of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  */
> > void deleteState(String key, String tenant, String ns, String name);
> >
> > /**
> >  * Delete the state value for the key, but don't wait for the
> > operation to be completed
> >  *
> >  * @param key   name of the key
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  */
> > CompletableFuture deleteStateAsync(String key, String tenant,
> > String ns, String name);
> >
> > /**
> >  * Increment the builtin distributed counter referred by key.
> >  *
> >  * @param keyThe name of the key
> >  * @param amount The amount to be incremented
> >  * @param tenant the state tenant name
> >  * @param ns the state namespace name
> >  * @param name the state store name
> >  */
> > void incrCounter(String key, long amount, String tenant, String ns,
> > String name);
> >
> >  

Re: [DISCUSS] Add icebox label for issues and PRs that have been inactive for more than 4 weeks

[Enrico Olivelli]

I am +1 with the initiative.

I would like to add a suggestion, maybe I am exaggerating...
Why not "closing" those PRs ?
Closing a PR does not mean to delete it

btw I am fine with the process you suggested

Enrico

Il giorno mer 12 gen 2022 alle ore 18:54 Michael Marshall
 ha scritto:
>
> > Ok, both "status/stale" and "status/inactive” looks good. Let's use
> > "status/inactive”
>
> +1 - I agree with using "status/inactive" for these issues/PRs.
>
> >> Can the time period be made a configuration parameter to make it easy to
> adjust?
> >Yes, we can easy to change the CI params.
>
> I agree with setting it to 4 weeks as an initial value, and it's good
> that it'll be easily tunable.
>
> Overall, +1 for adding automated labeling--I think it will help
> reviewers prioritize which issues/PRs they review.
>
> Thanks for moving this discussion forward, Penghui.
>
> Thanks,
> Michael
>
> On Wed, Jan 12, 2022 at 11:04 AM PengHui Li  wrote:
> >
> > > I used the "status/stale" label for some old PRs that I closed.
> >
> > I think that "status/inactive” would be a more descriptive label than
> > “icebox”.
> >
> > Ok, both "status/stale" and "status/inactive” looks good. Let's use
> > "status/inactive”
> >
> > > Can the time period be made a configuration parameter to make it easy to
> > adjust?
> >
> > Yes, we can easy to change the CI params.
> >
> > Thank you Dave for the quick response.
> >
> > Penghui
> >
> > On Thu, Jan 13, 2022 at 12:48 AM Dave Fisher  wrote:
> >
> > >
> > > > On Jan 12, 2022, at 8:15 AM, PengHui Li  wrote:
> > > >
> > > > Hi Pulsar Community,
> > > >
> > > > I want to start a discussion about introducing an icebox label that can
> > > be
> > > > added to
> > > > the issue or PR by pulsar bot automatically to help us can focus on the
> > > > active PRs
> > > > and issue. To avoid missing merge PRs, review PRs, triage issues.
> > >
> > > I used the "status/stale" label for some old PRs that I closed.
> > >
> > > I think that "status/inactive” would be a more descriptive label than
> > > “icebox”.
> > >
> > > >
> > > > It looks like the following:
> > > >
> > > > 1. If the issue or PR is inactive for more than 4 weeks, the pulsar bot
> > > add
> > > > the icebox label
> > > > 2. If the issue or PR is re-active again, the pulsar bot remove the
> > > icebox
> > > > label
> > > >
> > > > How to determine the PR or issue is inactive?
> > > >
> > > > 1. No comments for 4 weeks.
> > > > 2. No code review(approve, comment, or change request) for 4 weeks.
> > > > 3. No commits for 4 weeks.
> > > > 4. No description update for 4 weeks.
> > >
> > > Can the time period be made a configuration parameter to make it easy to
> > > adjust?
> > >
> > > >
> > > > How to determine the PR or issue is re-inactive?
> > > >
> > > > With the icebox label first and:
> > > >
> > > > 1. New comment added
> > > > 2. New commits pushed
> > > > 3. Description updated
> > > > 4. New code review updates
> > > >
> > > > Note: all the approved PRs we should not add the icebox label
> > > >
> > > > This will help us to focus on the active issues and PRs so that we can
> > > > track the active issues and PRs better first. After we get this part 
> > > > done
> > > > (maybe keep active opened PR under 20 and active opened issue under 
> > > > 50?),
> > > > we can move forward to continue to handle the stale PRs (already
> > > discussed
> > > > in https://lists.apache.org/thread/k7lyw0q0fyc729w0fqlj5vqng5ny63f2).
> > >
> > > Great initiative!
> > >
> > > +1
> > >
> > > All the best,
> > > Dave
> > >
> > >
> > > >
> > > > Thanks,
> > > > Penghui
> > >
> > >

Re: [DISCUSSION] PIP-135: Include MetadataStore backend for Etcd

[Enrico Olivelli]

+1

Enrico

Il Gio 13 Gen 2022, 06:01 Joe F  ha scritto:

> +1
>
> On Wed, Jan 12, 2022 at 3:52 PM Aloys Zhang  wrote:
>
> > +1
> >
> > 陳智弘  于2022年1月12日周三 10:19写道：
> >
> > > +1
> > >
> > > Haiting Jiang  於 2022年1月12日 週三 09:50 寫道：
> > >
> > > > +1
> > > >
> > > > On 2022/01/12 01:44:21 PengHui Li wrote:
> > > > > +1
> > > > >
> > > > > Penghui
> > > > >
> > > > > On Wed, Jan 12, 2022 at 8:39 AM mattison chao <
> > mattisonc...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Wed, 12 Jan 2022 at 08:09, Matteo Merli 
> > > wrote:
> > > > > >
> > > > > > > https://github.com/apache/pulsar/issues/13717
> > > > > > >
> > > > > > > -
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > Since all the pieces that composed the proposal in PIP-45 were
> > > > finally
> > > > > > > merged
> > > > > > > and are currently ready for 2.10 release, it is now possible to
> > add
> > > > other
> > > > > > > metadata backends that can be used to support a BookKeeper +
> > Pulsar
> > > > > > > cluster.
> > > > > > >
> > > > > > > One of the popular systems that is most commonly used as an
> > > > alternative
> > > > > > of
> > > > > > > ZooKeeper is Etcd, thus it makes sense to have this as the
> first
> > > > > > > non-zookeeper
> > > > > > > implementation.
> > > > > > >
> > > > > > > ## Goal
> > > > > > >
> > > > > > > Provide an Etcd implementation for the `MetadataStore` API.
> This
> > > will
> > > > > > allow
> > > > > > > users to deploy Pulsar clusters using Etcd service for the
> > metadata
> > > > and
> > > > > > it
> > > > > > > will
> > > > > > > not require the presence of ZooKeeper.
> > > > > > >
> > > > > > >
> > > > > > > ## Implementation
> > > > > > >
> > > > > > >  * Use the existing JEtcd Java client library for Etcd
> > > > > > >  * Extends the `AbstractBatchedMetadataStore` class, in order
> to
> > > > reuse
> > > > > > the
> > > > > > >transparent batching logic that will be shared with the
> > > ZooKeeper
> > > > > > >implementation.
> > > > > > >
> > > > > > > Work in progress: https://github.com/apache/pulsar/pull/13225
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Matteo Merli
> > > > > > > 
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd

[Enrico Olivelli]

+1 (binding)

Thanks
Enrico

Il Ven 14 Gen 2022, 23:52 Matteo Merli  ha scritto:

> https://github.com/apache/pulsar/issues/13717
>
> -
>
> ## Motivation
>
> Since all the pieces that composed the proposal in PIP-45 were finally
> merged
> and are currently ready for 2.10 release, it is now possible to add other
> metadata backends that can be used to support a BookKeeper + Pulsar
> cluster.
>
> One of the popular systems that is most commonly used as an alternative of
> ZooKeeper is Etcd, thus it makes sense to have this as the first
> non-zookeeper
> implementation.
>
> ## Goal
>
> Provide an Etcd implementation for the `MetadataStore` API. This will allow
> users to deploy Pulsar clusters using Etcd service for the metadata and it
> will
> not require the presence of ZooKeeper.
>
>
> ## Implementation
>
>  * Use the existing JEtcd Java client library for Etcd
>  * Extends the `AbstractBatchedMetadataStore` class, in order to reuse the
>transparent batching logic that will be shared with the ZooKeeper
>implementation.
>
> Work in progress: https://github.com/apache/pulsar/pull/13225
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] PIP-124: Pulsar Client Shared State API

[Enrico Olivelli]

@matteo ping

Enrico

Il Mer 29 Dic 2021, 08:35 Enrico Olivelli  ha scritto:

> Matteo,
>
> Il Mer 29 Dic 2021, 02:57 Matteo Merli  ha
> scritto:
>
>> > * Add an API to the Java client that makes it easier to maintain a
>> consistent Share State between instances of an application.
>> > * Provide some ready to use recipes, like a simple key-value store
>> >
>> > It is not a goal to implement a Pulsar backed Database system
>>
>> While the first use case for Pulsar was indeed to be the
>> messaging/replication platform for a distributed database, and it has
>> been working in production for many years, I'm not convinced to add
>> this level of API as part of the Pulsar client API.
>>
>> Pulsar API has been designed to be high-level and easy to use (and
>> reason about), with in mind the use cases of application developers. I
>> don't think that a "storage" level API fits well with the rest of
>> abstractions.
>>
>> > public interface PulsarMap extends AutoCloseable {
>> > ..
>> >  CompletableFuture put(K key, V value)
>>
>> If all the logic is implemented in the client side, when there are
>> multiple clients sharing the same, how can any of them mutate the
>> state, since we actually enforce that there is a single exclusive
>> producer? Would a user get an error if there's already a different
>> client writing?
>>
>> My impression is that, while looking convenient, a shared Map
>> interface is not the best abstraction for either case:
>>  * If you're actually building a DB, you will definitely need access
>> to the log itself rather than a Map interface
>>  * If you want to share some state across multiple processes without
>> using a DB, there are many tricky API, consistency and semantic
>> problems to solve, many of which are just pushed down to the
>> application which will need to be aware and understand them. At that
>> point, I would seriously recommend using a DB, or if the question is:
>> "I don't want to use an additional external system", then to use the
>> BK TableService component.
>>
>
> This is usually not a option because the BK TableService does not support
> well multi tenancy and also the application would need to connect to the
> Bookies (think about configuration, security...)
>
>
>>
>> I think this feature should be best viewed as a recipe, as it doesn't
>> depend on or benefits from any internal broker support. If there are
>> enough interest and concrete use cases it can be then included later.
>>
>
> My initial proposal was to push this to Pulsar Adapters.
> I changed the proposal before sending the PIP because I think it very
> useful for Protocol Handlers  and in Pulsar IO connectors.
>
> I am totally fine to add this to pulsar-adapters, but I want to see this
> in the Pulsar repo and released as part of an official Pulsar recipe.
>
> @Matteo does this sound like a good option to you?
>
> Otherwise we miss the possibility to make it easier for Pulsar users to
> leverage this power of Pulsar.
>
> In Pravega you have State Synchronizers and they are a great foundational
> API and we are missing something like that in Pulsar.
>
> Enrico
>
>
>
>> --
>> Matteo Merli
>> 
>>
>> On Fri, Dec 24, 2021 at 1:53 AM Enrico Olivelli 
>> wrote:
>> >
>> > Hello everyone,
>> > I want to start a discussion about PIP-124 Pulsar Client Shared  State
>> API
>> >
>> > This is the PIP document
>> > https://github.com/apache/pulsar/issues/13490
>> >
>> > This is a demo implementation (a proof-of-concept):
>> > https://github.com/eolivelli/pulsar-shared-state-manager
>> >
>> > Please take a look and share your thoughts
>> >
>> > I believe that this will unlock the potential of the Exclusive
>> > Producer and it will also make easier the life of many developers who
>> > are using Pulsar and need some API to share configuration, metadata,
>> > or any simple key-value data structure without adding a Database or
>> > other components to their library, Pulsar IO connector or Pulsar
>> > Protocol Handler.
>> >
>> > Thanks
>> > Enrico
>>
>

Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd

[Enrico Olivelli]

Il Sab 15 Gen 2022, 09:10 tamer Abdlatif  ha
scritto:

> Will that affect the existing ZK metadata in a pulsar instance , When we
> upgrade from a lower version to 2.10 ?  In other words,
> Do we need a metadate migration to switch from ZK to Etcd ?
>

There is no need to migrate.

Most probably the first release will bring this feature as non production
ready and it will take some time to stabilise

Enrico



>
> Thanks
> Tamer
>
>
>
> On Fri, 14 Jan 2022, 22:52 Matteo Merli,  wrote:
>
> > https://github.com/apache/pulsar/issues/13717
> >
> > -
> >
> > ## Motivation
> >
> > Since all the pieces that composed the proposal in PIP-45 were finally
> > merged
> > and are currently ready for 2.10 release, it is now possible to add other
> > metadata backends that can be used to support a BookKeeper + Pulsar
> > cluster.
> >
> > One of the popular systems that is most commonly used as an alternative
> of
> > ZooKeeper is Etcd, thus it makes sense to have this as the first
> > non-zookeeper
> > implementation.
> >
> > ## Goal
> >
> > Provide an Etcd implementation for the `MetadataStore` API. This will
> allow
> > users to deploy Pulsar clusters using Etcd service for the metadata and
> it
> > will
> > not require the presence of ZooKeeper.
> >
> >
> > ## Implementation
> >
> >  * Use the existing JEtcd Java client library for Etcd
> >  * Extends the `AbstractBatchedMetadataStore` class, in order to reuse
> the
> >transparent batching logic that will be shared with the ZooKeeper
> >implementation.
> >
> > Work in progress: https://github.com/apache/pulsar/pull/13225
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [ANNOUNCE] New PMC Member - Lari Hotari

[Enrico Olivelli]

Congratulations!



Enrico

Il Lun 17 Gen 2022, 22:03 frank  ha scritto:

> Congrats Lari - very well deserved
>
> Frank
>
>
>
>
>
>
>  On Mon, 17 Jan 2022 15:50:37 -0500 w...@apache.org
> wrote 
>
> Hi -
>
> The Apache Pulsar Project Management Committee (PMC) has invited Lari
> Hotari
> (https://github.com/lhotari) as a member of the PMC and we are pleased to
> announce that he has accepted.
>
> He is very active in the community on GitHub and on the pulsar slack
> channel.
>
> Welcome Lari to the Apache Pulsar PMC
>
> Best Regards,
> Dave Fisher on behalf of the Pulsar PMC
>
>
>

Re: [DISCUSS] PIP-86: Pulsar Functions: Preload and release external resources

[Enrico Olivelli]

Neng,

Il Gio 20 Gen 2022, 20:20 Neng Lu  ha scritto:

> Hi All,
>
> Just want to bring this PIP[1] to your attention. The PRs [2] have been
> open for quite some time. The feature is valuable for many use cases and I
> would like to help the original author to push the effort on it.
>
> The general idea is introducing a new API for Pulsar Functions which allows
> develop to customize some setup and close logic.


I am +1  on your proposal.
I left some feedback on the second PR. Basically we need some unit tests
and integration tests.
The first PR looks obsolete, please close it.

Enrico

The API should look like
> this:
>
> ```
> public interface RichFunction extends Function{
>
> /**
>  * Called when function instance start
>  *
>  * @throws Exception
>  */
> void setup(Context context) throws Exception;
>
> /**
>  * Called when function instance close
>  *
>  * @throws Exception
>  */
> void tearDown() throws Exception;
> }
> ```
>
> Please join the discussion if you have any questions or concerns for this
> new API.
>
> [1] PIP-86
> <
> https://github.com/apache/pulsar/wiki/PIP-86%3A-Pulsar-Functions%3A-Preload-and-release-external-resources
> >
> [2] PR-2  PR-13205
> 
>
> Best regards,
> Neng
>

Re: [ANNOUNCE] New Committer: Haiting Jiang

[Enrico Olivelli]

Congratulations!


Enrico

Il Ven 21 Gen 2022, 08:38 Aloys Zhang  ha scritto:

> Congratulations!
>
> zhangao  于2022年1月21日周五 15:07写道：
>
> > Congratulations! 
> >
> >
> > Best Regards,
> > zhangao
> > -- 原始邮件 --
> > 发件人:
> >   "dev"
> > <
> > zhai...@apache.org>;
> > 发送时间: 2022年1月21日(星期五) 下午2:59
> > 收件人: "Dev" > us...@pulsar.apache.org>;
> >
> > 主题: [ANNOUNCE] New Committer: Haiting Jiang
> >
> >
> >
> > The Apache Pulsar Project Management Committee (PMC) has invited Haiting
> > Jiang
> > (https://github.com/jason918) to become a committer and we are pleased
> to
> >
> > announce that he has accepted.
> >
> >
> > Haiting contributed a lot of interesting additions to the project, and
> > helped a lot to answer the user’s questions like in github issues, wechat
> > groups and slack channels.  His full list of contributions in github
> > can be
> > found at https://github.com/apache/pulsar/commits?author=jason918.
> >
> >
> > Welcome and Congratulations, Haiting! Please enjoy the journey as a
> > committer :)
> >
> >
> > Please join us in congratulating and welcoming Haiting onboard!
> >
> > Best Regards,
> > Jia on behalf of the Pulsar PMC
>

Re: [VOTE] PIP-86: Pulsar Functions: Preload and release external resources

[Enrico Olivelli]

+1

Enrico

Il Ven 21 Gen 2022, 21:18 Jerry Peng  ha
scritto:

> +1
>
> On Fri, Jan 21, 2022 at 12:07 PM Neng Lu  wrote:
>
> > Hi All,
> >
> > I would like to start a VOTE on the PIP 86. (If it's already been voted,
> > please let me know.)
> >
> > The issue for PIP 86 is here:
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-86%3A-Pulsar-Functions%3A-Preload-and-release-external-resources
> > And the initial implementation is here:
> > https://github.com/apache/pulsar/pull/13205
> >
> > Please VOTE within 48 hours.
> >
> > Best Regards,
> > Neng Lu
> >
>

Re: [LAZY CONSENSUS] Sending 2.6 release line to EOF

[Enrico Olivelli]

Good.
I will send an email to u...@pulsar.apache.org that states the EOL
status for 2.6

Enrico

Il giorno lun 10 gen 2022 alle ore 19:26 Chris Herzog
 ha scritto:
>
> I agree with 2.6 EOL.
>
> On Mon, Jan 10, 2022 at 2:35 AM Enrico Olivelli  wrote:
>
> > This thread is now to moving branch-2.6 (the Pulsar 2.6 release line)
> > to End-Of-Life status
> >
> > What EOL means here:
> > - No more releases, even for security bugs
> > - No cherry-picks to branch-2.6
> >
> > We will apply the Lazy Consensus rules
> > https://www.apache.org/foundation/glossary.html#LazyConsensus
> >
> > If no-one objects within 72 hours I will send an official announcement
> > of the EOL status of 2.6.
> >
> > Enrico
> >
> >
> > Il giorno mar 4 gen 2022 alle ore 06:04 Michael Marshall
> >  ha scritto:
> > >
> > > I agree with Dave. Lazy consensus gives us exactly what we need: to
> > > know if someone would like branch-2.6 to stay alive. If no one
> > > responds that they need it within the time period, then no one needs
> > > it.
> > >
> > > PIP 47 briefly mentions our EOL policy, but the policy doesn't cover
> > > our current case where we released fewer than 4 minor versions in a
> > > year. I think we'd benefit from clarifying and improving our
> > > documentation for our EOL policy.
> > >
> > > > This project VOTEs on more things than any Apache project I’ve been
> > involved with during the last 14 years.
> > >
> > > The new PIP process, with its increased purview, has increased the
> > > number of votes. I have appreciated the increased development
> > > discussion on the ML, but maybe the discussions (especially the ones
> > > that look like votes) and then the subsequent votes are leading to the
> > > high vote volume.
> > >
> > > We could update the PIP process so that DISCUSSIONS without any
> > > pushback during a certain time period do not need VOTES.
> > >
> > > Thanks,
> > > Michael
> > >
> > >
> > > On Mon, Jan 3, 2022 at 6:00 PM Dave Fisher  wrote:
> > > >
> > > > This project VOTEs on more things than any Apache project I’ve been
> > involved with during the last 14 years.
> > > >
> > > > For most projects lazy consensus applies:
> > https://www.apache.org/foundation/glossary.html#LazyConsensus
> > > >
> > > > We also tend to “vote” during discussions which does not really
> > advance a conversation.
> > > >
> > > > If we want to have a VOTE then there are good reasons to start it NOW.
> > > >
> > > > All the Best,
> > > > Dave
> > > >
> > > > > On Jan 3, 2022, at 3:48 PM, Sijie Guo  wrote:
> > > > >
> > > > > Agree to have a vote to keep a record.
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, Jan 3, 2022 at 3:40 PM 陳智弘  wrote:
> > > > >
> > > > >> I think having a vote and quickly announce the EOF of 2.6.x will be
> > better
> > > > >> to the community.
> > > > >>
> > > > >>
> > > > >> Dave Fisher  於 2022年1月4日 週二 06:06 寫道：
> > > > >>
> > > > >>> I don’t think we need a VOTE. Let’s do this by LAZY CONSENSUS.
> > > > >>>
> > > > >>>> On Jan 3, 2022, at 10:05 AM, Enrico Olivelli  > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>> Hello,
> > > > >>>> We are no more cutting releases out of branch-2.6
> > > > >>>>
> > > > >>>> We must declare 2.6 EOF.
> > > > >>>>
> > > > >>>> IIUC there is no process established for this at the moment in the
> > > > >> Pulsar
> > > > >>>> project.
> > > > >>>>
> > > > >>>> So for this time I will call out a VOTE next week, as we did not
> > cut
> > > > >>>> releases for a few important bugs.
> > > > >>>>
> > > > >>>> Thoughts?
> > > > >>>>
> > > > >>>> Enrico
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> >
>
>
> --
>
>
> Chris Herzog Messaging Team | O 630 300 7718 | M 815 263 3764 |
> www.tibco.com
>
> <http://www.tibco.com/>

[ANNOUNCE] Apache Pulsar 2.6 (and previous release lines) is EOL

[Enrico Olivelli]

Hello,
This message is to formally announce the 2.6 release line, together
with all of the previous release lines (2.5, 2.4...) reached
End-Of-Line status.

This basically this means that:
- we are not going to cut new releases for 2.6
- users are encouraged to move to 2.7 or later

Best regards
Enrico Olivelli

Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires

[Enrico Olivelli]

What's the status of this PIP ?

We have a pending PR,
we can commit it as soon as the VOTE thread is officially closed
https://github.com/apache/pulsar/pull/13341


Il giorno gio 23 dic 2021 alle ore 01:17 Sijie Guo
 ha scritto:
>
> +1
>
> On Tue, Dec 21, 2021 at 3:22 PM Matteo Merli  wrote:
>
> > https://github.com/apache/pulsar/issues/13304
> >
> > Following the discussion, I have updated the proposal to also include
> > the deprecation and renaming of the config setting name to
> > `metadataSessionExpiredPolicy`.
> >
> >
> >
> > ---
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and 2.9
> > releases, it enabled the concept of re-acquirable resource locks and leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> > operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled to stabilize in 2.9 the configuration setting that
> > allows
> > brokers to continue operating in a safe mode when the session with
> > ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data plane
> > will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> > any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> > thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Deprecate the old setting and rename it to
> > `metadataSessionExpiredPolicy`, with default value set to `reconnect`.
> >

Re: Note for pulsar-client checkstyle PR

[Enrico Olivelli]

Yunze,

Il giorno mar 25 gen 2022 alle ore 13:14 Yunze Xu
 ha scritto:
>
> Hi all (especially committers),
>
> I opened a PR to enable checkstyle plugin for pulsar-client module just now.
> See https://github.com/apache/pulsar/pull/13940 
> .


Great work.
>
> It’s really a huge PR but I hope it could be included in Pulsar 2.10 release

+1

> because pulsar-client is one of the most huge modules of Pulsar. And the
> code style has been out of control for a long time.
>
> For Pulsar committers, please be careful to merge PRs that modify code in
> pulsar-client module after this PR is merged because it could make master
> branch broken.

I am not sure if there is a way to revalidate the PRs.
IIUC when you close/reopen a PR it is not merged again with latest
master before rerunning CI

Enrico

>
> Thanks,
> Yunze

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

[Enrico Olivelli]

great work Lari !

what about upgrading to 2.9.1 and not to 2.8.2 ?
We are VOTing for 2.9.2 and 2.10 will be shipped soon

isn't 2.8.2 quite old at this point ?

Enrico

Il giorno mer 26 gen 2022 alle ore 14:52 Lari Hotari
 ha scritto:
>
> UPDATE:
> There's now a workaround for the issue with Zookeeper in Pulsar 2.8.x .
>
> The fix was to add timeout handling for the Zookeeper probes, the PR is 
> https://github.com/apache/pulsar-helm-chart/pull/214.
> An earlier PR https://github.com/apache/pulsar-helm-chart/pull/179 added 
> probe timeouts, but this only works since Kubernetes 1.20.
>
> Please proceed to review https://github.com/apache/pulsar-helm-chart/pull/190 
> which upgrades Pulsar images to 2.8.2 and bumps the Chart version to 2.8.0 .
>
> The change to make Pulsar image version default to Chart's appVersion is in a 
> separate PR https://github.com/apache/pulsar-helm-chart/pull/200 . That 
> change was earlier requested by Sijie in this email thread. I'd appreciate 
> feedback and reviews on that change too.
>
> I'd like to suggest that we go ahead in reviewing and merging 
> https://github.com/apache/pulsar-helm-chart/pull/190 asap so that we can 
> finally move to Pulsar 2.8.x in the Pulsar Helm Chart.
>
> BR,
> Lari
>
>
> On 2022/01/12 12:52:47 Lari Hotari wrote:
> > Hi Sijie,
> >
> > Thanks for the suggestions.
> >
> > > That means:
> > >
> > > > 1. We should have a separate `version` from `appVersion`.
> > > > 2. We should use the Pulsar image version as the `appVersion`.
> > > > 3. It is okay to only update broker and proxy images version and leave
> > > zookeeper and bookkeeper version unchanged.
> > >
> >
> > I believe 1.) is already how we handle apache/pulsar-helm-chart. version of
> > the chart is independent of appVersion.
> > For 2.) I have created https://github.com/apache/pulsar-helm-chart/pull/200
> > . @si...@apache.org  is that what you meant? Please
> > review the PR.
> > 3.) I guess this is more about addition documentation? Are there any
> > changes that need to be made in the Apache Pulsar Helm Chart to support
> > this?
> >
> > BR,
> >
> > Lari
> >
> >
> > On Fri, Jan 7, 2022 at 2:58 AM Sijie Guo  wrote:
> >
> > > The fundamental problem I see is that we don't have a proper helm chart
> > > release and we don't have good versioning guidance.
> > >
> > > Chart should have its own version, which is independent of the Pulsar
> > > version.
> > >
> > > Also, we need to provide a guide to the community - most of the time, you
> > > don't need to upgrade zookeeper and bookkeeper. Because these two
> > > components are rarely changed.
> > >
> > > That means:
> > >
> > > 1. We should have a separate `version` from `appVersion`.
> > > 2. We should use the Pulsar image version as the `appVersion`.
> > > 3. It is okay to only update broker and proxy images version and leave
> > > zookeeper and bookkeeper version unchanged.
> > >
> > > That's probably not the final guide. But at least, we can have something 
> > > to
> > > get started to formalize the process. This guide will then help us handle
> > > such situations better.
> > >
> > > - Sijie
> > >
> > > On Wed, Jan 5, 2022 at 6:00 AM 陳智弘  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >   From my side, I think currently merging the request without regarding
> > > the
> > > > known issue and announcing this information on the website is a good
> > > > option.
> > > >
> > > > <
> > > >
> > > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> > > > >
> > > > 不含病毒。www.avg.com
> > > > <
> > > >
> > > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> > > > >
> > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> > > >
> > > > Lari Hotari  於 2022年1月5日 週三 下午9:04寫道：
> > > >
> > > > > Reminder: we need to decide about the developement of Apache Pulsar
> > > Helm
> > > > > chart.
> > > > > Please reply to the email or review
> > > > > https://github.com/apache/pulsar-helm-chart/pull/190 .
> > > > > That PR is blocked since a decision must be made whether it's fine to
> > > > make
> > > > > the change, although there's a known issue in Zookeeper when TLS is
> > > > > enabled.
> > > > > The issue is ZOOKEEPER-3988 /
> > > > > https://github.com/apache/pulsar/issues/11070 .
> > > > > The bug currently only impacts TLS since the change
> > > > > https://github.com/apache/pulsar-helm-chart/pull/180 switched
> > > > > to use NIOServerCnxnFactory for Zookeeper. NIOServerCnxnFactory 
> > > > > doesn't
> > > > > support TLS and the impacted NettyServerCnxnFactory must be used for
> > > TLS.
> > > > >
> > > > > How do we handle the decision? Can we proceed in merging
> > > > > https://github.com/apache/pulsar-helm-chart/pull/190 regardless of the
> > > > > known issue?
> > > > >
> > > > > BR,
> > > > > Lari
> > > > >
> > > > > On Mon, Jan 3, 2022 at 4:39 PM Lari Hotari  wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > There

Re: [VOTE] PIP-119: Enable consistent hashing by default on KeyShared dispatcher

[Enrico Olivelli]

HEADS up

I have found this problem:
https://github.com/apache/pulsar/issues/10750

We should fix it before changing the default value

Enrico

Il giorno gio 23 dic 2021 alle ore 06:37 Michael Marshall
 ha scritto:
>
> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:17 PM Sijie Guo  wrote:
> >
> > +1
> >
> > On Tue, Dec 21, 2021 at 3:23 PM Matteo Merli  wrote:
> >
> > > This is the voting thread for PIP-119. It will stay open for at least 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13305
> > >
> > > ---
> > >
> > > ## Motivation
> > >
> > > The consistent hashing implementation to uniformly assign keys to 
> > > consumers
> > > in the context of a KeyShared subscription, was introduced in
> > > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar
> > > 2.6.0.
> > >
> > > While consistent hashing can use slightly more memory in certain cases, it
> > > is
> > > more suitable as a general default implementation, as it leads to a fairer
> > > distribution of keys across consumers, and avoiding corner cases that
> > > depend
> > > on the sequence of addition/removal of consumers.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the setting:
> > >
> > > ```properties
> > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use
> > > splitting ranges or
> > > # consistent hashing to reassign keys to new consumers
> > > subscriptionKeySharedUseConsistentHashing=false
> > > ```
> > >
> > > Change its default value to `true`.
> > >
> > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still
> > > be
> > > able to use the old implementation.
> > >
> > > --
> > > Matteo Merli
> > > 
> > >

CVE-2021-41571: Apache Pulsar: Pulsar Admin API allows access to data from other tenants using getMessageById API

[Enrico Olivelli]

Severity: moderate

Description:

In Apache Pulsar it is possible to access data from BookKeeper that
does not belong to the topics accessible by the authenticated user.

The Admin API get-message-by-id requires the user to input a topic and
a ledger id. The ledger id is a pointer to the data, and it is
supposed to be a valid id for the topic.
Authorisation controls are performed against the topic name and there
is no proper validation that the ledger id is valid in the context of
such ledger.
So it may happen that the user is able to read from a ledger that
contains data owned by another tenant.

This issue affects Apache Pulsar Apache Pulsar version 2.8.0 and prior
versions; Apache Pulsar version 2.7.3 and prior versions; Apache
Pulsar version 2.6.4 and prior versions.

This issue is being tracked as https://github.com/apache/pulsar/issues/11814

Mitigation:

If you are running Pulsar behind a proxy you can disable access to the
REST API for the flawed API

/admin/v2/non-persistent/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}

References:

https://pulsar.apache.org/admin-rest-api/#operation/getLastMessageId
https://github.com/apache/pulsar/issues/11814

Re: [PR] CI workflow to check dependencies with OWASP

[Enrico Olivelli]

Great idea

I will review the PRs

Thanks
Enrico


Il Lun 31 Gen 2022, 21:33 Andrey Yegorov  ha
scritto:

> Hello,
>
> As a final step in the series of PRs to upgrade old dependencies with
> various CVEs (by Nicolo and I) I added a PR that introduces extra check on
> pom.xml files changes: it will run OWASP dependency check and fail if any
> CVE level >= 7 is detected.
>
> Please review this PR https://github.com/apache/pulsar/pull/13972 an one
> more pending fix from Nicolo: https://github.com/apache/pulsar/pull/13943
>
> There is an existing workflow that runs daily but it has limited visibility
> at the moment:
>
> https://github.com/apache/pulsar/actions/workflows/ci-owasp-dependency-check.yaml
> As a next step we can work on making it email the dev list when it fails
> but the check on PR when pom files change will be more immediately visible.
>
> Similar changes are pending for BookKeeper.
>
> --
> Andrey Yegorov
>

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

(sorry for the late reply, I am still testing, I had some other priorities).

I hope that the community will test this RC and report back


Enrico

Il giorno mar 25 gen 2022 alle ore 15:07 Ran Gao  ha scritto:
>
> Sorry, the 2.9.2 release candidate-1 has a wrong sign certificate.
>
> This is the second release candidate for Apache Pulsar, version 2.9.2.
>
> *** Please download, test, and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-2/
>
> SHA-512 checksums:
>
> 563f65582c5307b4ef1e0322958ed19d7c181fb8bb8d7b8cab06ab0a6adb5520f7d18b6f97960b93c3318815529a8b8721e00e9cc9484532a2e5ed3221450094
>  ./apache-pulsar-2.9.2-bin.tar.gz
> 60d1049611b938b0ddc769132124d43820728afc8a06813a5ec9efc095c5497c59d9bbcaaf7df5b0c0e97e051d66f59c1f8ee08885d05ca2c635773e0283770a
>  ./apache-pulsar-2.9.2-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1136
>
> The tag to be voted upon:
> v2.9.2-candidate-2 (8a5d2253b888b3b865a2aedf635d672821c7)
> https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.

Re: [PR] pulsar-adapters: remove Flink adapter

[Enrico Olivelli]

The patch is merged.


Thank you Andrey, removing dead code is a good thing, as it allows the
community to not waste energy.


Enrico


Il Ven 4 Feb 2022, 06:04 Devin Bost  ha scritto:

> I support this change. Having different connectors in each project is
> actually really confusing for new folks since it's not automatically clear
> which adapter they should be using.
>
> --
> Devin G. Bost
>
> On Thu, Feb 3, 2022, 1:17 PM Andrey Yegorov 
> wrote:
>
> > Hello,
> >
> > I created a PR to remove Flink from the Pulsar adapters project:
> > https://github.com/apache/pulsar-adapters/pull/34
> >
> > Motivation is:
> > * It is based on an old version of Flink (1.7.2, current is 1.14.3) which
> > brings in dependencies with various CVEs. 1.7 is not maintained anymore.
> > * Flink added a pulsar connector in their project
> >
> >
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-pulsar
> > * Previous releases of the adapter still available and can be used, if
> > needed
> >
> > Please review and let me know if there is a reason to not
> deprecate/remove
> > the Flink adapter.
> >
> > --
> > Andrey Yegorov
> >
>

Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format

[Enrico Olivelli]

What happens when you upgrade the Prometheus client ?

Can you share some examples of "before" and "after" ?
My understanding is that you posted how it looks like "after" the upgrade

Thanks for working on this

Enrico

Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
 ha scritto:
>
> Before, I am working on bumping Prometheus client to 0.12.0, but they
> introduce a breaking change,
> https://github.com/prometheus/client_java/pull/615, adopt the `OpenMetrics
> format`, which acquired all counters have `_total` suffix,
>
> but our metrics now have these metrics, there are not satisfied with the
> OpenMetrics format, for example:
>
> - pulsar_connection_closed_total_count
>
> - pulsar_connection_created_total_count
>
> - pulsar_source_received_total_1min
>
> - system_exceptions_total_1min
>
>
> I want to discuss, Should we adapt the `OpenMetrics format`?
>
> If we want to be compatible with Open Metrics, I suggest adding metrics
> named `_total` in a release version like 2.10.0, and removing the origin
> metric in the next release like 2.11.0.

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

Penghui, Gao,
see my comments below please

Il giorno mar 8 feb 2022 alle ore 09:16 Hang Chen
 ha scritto:
>
> +1 (binding)
>
> Verified the following context.
> 1. Checked the checksum and licenses
> 2. Build from the source code successfully
> 3. Start standalone and run pulsar-perf with produce and consume
> 4. Checked the Pulsar function and stateful functions
> 5. Run Pulsar with KOP and publish and consume successfully with Kafka
> 3.1.0 client
>
> Best,
> Hang
>
> PengHui Li  于2022年2月7日周一 18:01写道：
> >
> > +1 (binding)
> >
> > 1. Checked the signature
> > 2. Build from the source successfully
> > 3. Start standalone
> > 4. Publish and consume successfully
> > 5. Cassandra connect works well
> > 6. Checked state function
> >
> > And passed our internal integration tests.
> >
> > Regards,
> > Penghui
> >
> > On Mon, Feb 7, 2022 at 4:37 PM PengHui Li  wrote:
> >
> > > It's not a regression in 2.9.2, we should not block the 2.9.2 release.
> > > Instead, we can have the fix in 2.9.3.

It depends on the timeline of 2.9.3...but we cannot "promise" we will
follow up immediately with another release.

My understanding is that now if you enable transactions then you
cannot use regular Pulsar producers.
And from 2.9 we said that Transactions are no more BETA

This is very bad from my point of view.

I agree that this is not a regression of 2.9.2 vs 2.9.1, but basically
the Transactions feature is not usable

There is already a PR open, with a good discussion.
I believe that we should not hurry up in doing this release and we can
fix the problem.

We should fix the problem and then roll a new RC

If my understanding is correct, then I am -1 in releasing this RC.
If it is not correct, then I am happy to continue testing this RC.

Enrico




> > >
> > > Penghui
> > >
> > > On Thu, Feb 3, 2022 at 8:42 PM Nicolò Boschi  wrote:
> > >
> > >> Hi Ran, thanks for driving the release.
> > >>
> > >> I haven't tested the rc yet but I firmly believe we should include this
> > >> pull [1] which fixes a regression introduced in Pulsar 2.9.0
> > >>
> > >> [1] https://github.com/apache/pulsar/pull/14097
> > >>
> > >>
> > >>
> > >> Il giorno mer 2 feb 2022 alle ore 08:34 Enrico Olivelli <
> > >> eolive...@gmail.com>
> > >> ha scritto:
> > >>
> > >> > (sorry for the late reply, I am still testing, I had some other
> > >> > priorities).
> > >> >
> > >> > I hope that the community will test this RC and report back
> > >> >
> > >> >
> > >> > Enrico
> > >> >
> > >> > Il giorno mar 25 gen 2022 alle ore 15:07 Ran Gao  ha
> > >> > scritto:
> > >> > >
> > >> > > Sorry, the 2.9.2 release candidate-1 has a wrong sign certificate.
> > >> > >
> > >> > > This is the second release candidate for Apache Pulsar, version 
> > >> > > 2.9.2.
> > >> > >
> > >> > > *** Please download, test, and vote on this release. This vote will
> > >> stay
> > >> > > open
> > >> > > for at least 72 hours ***
> > >> > >
> > >> > > Note that we are voting upon the source (tag), binaries are provided
> > >> for
> > >> > > convenience.
> > >> > >
> > >> > > Source and binary files:
> > >> > >
> > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-2/
> > >> > >
> > >> > > SHA-512 checksums:
> > >> > >
> > >> > >
> > >> >
> > >> 563f65582c5307b4ef1e0322958ed19d7c181fb8bb8d7b8cab06ab0a6adb5520f7d18b6f97960b93c3318815529a8b8721e00e9cc9484532a2e5ed3221450094
> > >> > >  ./apache-pulsar-2.9.2-bin.tar.gz
> > >> > >
> > >> >
> > >> 60d1049611b938b0ddc769132124d43820728afc8a06813a5ec9efc095c5497c59d9bbcaaf7df5b0c0e97e051d66f59c1f8ee08885d05ca2c635773e0283770a
> > >> > >  ./apache-pulsar-2.9.2-src.tar.gz
> > >> > >
> > >> > > Maven staging repo:
> > >> > >
> > >> https://repository.apache.org/content/repositories/orgapachepulsar-1136
> > >> > >
> > >> > > The tag to be voted upon:
> > >> > > v2.9.2-candidate-2 (8a5d2253b888b3b865a2aedf635d672821c7)
> > >> > > https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-2
> > >> > >
> > >> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > >> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >> > >
> > >> > > Please download the source package, and follow the README to build
> > >> > > and run the Pulsar standalone service.
> > >> >
> > >>
> > >>
> > >> --
> > >> Nicolò Boschi
> > >>
> > >

Re: [DISCUSS] Apache Pulsar 2.10.0 release

[Enrico Olivelli]

PengHui,
There is a recent discussion with Matteo (at the community meetings)
about preparing the release branch a couple of weeks before sending
out the official VOTE.

What about creating the branch-2.10 as soon as possible?
We will commit to that branch only the fixes needed to make 2.10.0 stable
This way we will be free to commit new stuff to master branch without
impacting the stability of 2.10

This way people can start validating 2.10 seriously in order to catch
problems before sending out the RC

Does it sound like a good idea to you ?
Enrico

Il giorno mer 9 feb 2022 alle ore 09:25 PengHui Li
 ha scritto:
>
> Hi all,
>
> Sorry for the late reply, due to my vacation these days, we got a delay
> here.
>
> Most of the changes of 2.10.0 are getting merged, for now, there are 14
> opened PRs(10 approved)
> https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+milestone%3A2.10.0
>
> I will take care of them and try to get them merged.
> After the above PRs get merged, I will build the release and start the vote.
> Please let me know if you have any questions about the 2.10.0 release.
> And, also looking forward to more people taking a look at the opened PRs.
>
> Regards,
> Penghui
>
>
>
>
> On Tue, Jan 4, 2022 at 7:56 AM Sijie Guo  wrote:
>
> > +1.
> >
> > All make sense to me!
> >
> > We probably need to move to the feature frozen stage in order to cut a
> > release at the end of January.
> >
> > - Sijie
> >
> > On Sun, Dec 26, 2021 at 8:46 PM PengHui Li  wrote:
> >
> > > Hi, everyone
> > >
> > > I hope you’ve all been doing well. I would like to start an email thread
> > to
> > > discuss features that we planned for 2.10.0.
> > > According to the time-based release plan
> > > https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan,
> > > we should release 2.10.0 at the end of December 2021, since we have
> > reached
> > > the end of December,
> > > I would like to target the 2.10.0 to the end of January 2022
> > >
> > > There are some powerful features and enhancements in 2.10.0 such as
> > >
> > > - PIP 84: Message redelivery epoch
> > > - PIP 104: Add new consumer type: TableView
> > > - PIP 106: Negative acknowledgment backoff
> > > - PIP 110: Topic customized metadata support
> > > - PIP 117: Change Pulsar standalone defaults
> > > - PIP 118: Do not restart brokers when ZooKeeper session expires
> > > - PIP 119: Enable consistent hashing by default on KeyShared dispatcher
> > > - PIP 120: Enable client memory limit by default
> > > - PIP 121: Pulsar cluster level auto failover
> > > - PIP 123: Pulsar metadata CLI tool
> > > - Metadata service batch operations
> > > - RocksDB metadata service backend
> > > - Etcd metadata service backend
> > > - Ack timeout redelivery backoff policy
> > > - Global topic policies
> > >
> > > Most of them have been completed, some work in progress we need to try to
> > > complete within 2 weeks.
> > > This can give me a 2 week buffer period to prepare for release and
> > complete
> > > the release vote.
> > > For the unfinished parts, we can move them to 2.11.0.
> > >
> > > Some proposals are just being discussed, so I do not list them because
> > I'm
> > > not sure if we can complete them in two weeks.
> > >
> > > You can find all the change lists from
> > >
> > >
> > https://github.com/apache/pulsar/pulls?q=milestone%3A2.10.0+-label%3Arelease%2F2.9.1
> > > There are more than 500 commits.
> > >
> > > If I missed something or you have any suggestions please let me know.
> > >
> > > Regards,
> > > Penghui
> > >
> >

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

are some other ongoing transaction fixes, we can't wait for all of
> > > them to be completed
> > > Please move them to 2.9.3 and don't block the 2.9.2 release.
> > >
> > > And https://github.com/apache/pulsar/pull/14097 also in the discussion
> > > stage,
> > > We should keep calm at this time, no need to hurry to merge a 100% clear
> > > plan,
> > > otherwise, we might introduce other regression in 2.9.2.
> > >
> > > Another point is non-transaction users are much larger than transaction
> > > users for now,
> > > there are many critical issue fixes in 2.9.2, I see many users can't wait
> > > for our release,
> > > they build from the branch-2.9 manually.
> > >
> > > We should follow the time-based release mode. If the release doesn't have
> > > critical security issues, regressions.
> > > we should not block the release, instead, we should prepare for the
> > release
> > > of 2.9.3 earlier.
> > >
> > > If the time-based release doesn't work, I think we should have a
> > discussion
> > > in the private email list
> > > to find a good solution for Pulsar release, Let everyone keep consistent
> > on
> > > the rules for release.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Tue, Feb 8, 2022 at 5:04 PM Enrico Olivelli 
> > wrote:
> > >
> > > > Penghui, Gao,
> > > > see my comments below please
> > > >
> > > > Il giorno mar 8 feb 2022 alle ore 09:16 Hang Chen
> > > >  ha scritto:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Verified the following context.
> > > > > 1. Checked the checksum and licenses
> > > > > 2. Build from the source code successfully
> > > > > 3. Start standalone and run pulsar-perf with produce and consume
> > > > > 4. Checked the Pulsar function and stateful functions
> > > > > 5. Run Pulsar with KOP and publish and consume successfully with
> > Kafka
> > > > > 3.1.0 client
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > PengHui Li  于2022年2月7日周一 18:01写道：
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > 1. Checked the signature
> > > > > > 2. Build from the source successfully
> > > > > > 3. Start standalone
> > > > > > 4. Publish and consume successfully
> > > > > > 5. Cassandra connect works well
> > > > > > 6. Checked state function
> > > > > >
> > > > > > And passed our internal integration tests.
> > > > > >
> > > > > > Regards,
> > > > > > Penghui
> > > > > >
> > > > > > On Mon, Feb 7, 2022 at 4:37 PM PengHui Li 
> > wrote:
> > > > > >
> > > > > > > It's not a regression in 2.9.2, we should not block the 2.9.2
> > > > release.
> > > > > > > Instead, we can have the fix in 2.9.3.
> > > >
> > > > It depends on the timeline of 2.9.3...but we cannot "promise" we will
> > > > follow up immediately with another release.
> > > >
> > > > My understanding is that now if you enable transactions then you
> > > > cannot use regular Pulsar producers.
> > > > And from 2.9 we said that Transactions are no more BETA
> > > >
> > > > This is very bad from my point of view.
> > > >
> > > > I agree that this is not a regression of 2.9.2 vs 2.9.1, but basically
> > > > the Transactions feature is not usable
> > > >
> > > > There is already a PR open, with a good discussion.
> > > > I believe that we should not hurry up in doing this release and we can
> > > > fix the problem.
> > > >
> > > > We should fix the problem and then roll a new RC
> > > >
> > > > If my understanding is correct, then I am -1 in releasing this RC.
> > > > If it is not correct, then I am happy to continue testing this RC.
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > >
> > > > > > >
> > > > > > > Penghui
> >

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

Gao,
The patch https://github.com/apache/pulsar/pull/14192 has been merged.
I believe that is better to roll out a new RC and have Pulsar 2.9.2 released


Thanks for the good discussion

Enrico

Il giorno gio 10 feb 2022 alle ore 05:11 PengHui Li
 ha scritto:
>
> It should not be a blocking issue, as I mentioned before we can work around
> and the issue happens for specific conditions
> And there are some other ongoing transaction fixes, for transactions using
> the latest branch-2.9 is the best option, not 2.9.2 even contain
> https://github.com/apache/pulsar/pull/14097
>
> So I suggest moving forward with the 2.9.2 release and focusing on the
> transaction fixes in 2.9.3.
>
> Regards,
> Penghui
>
> On Thu, Feb 10, 2022 at 11:02 AM Lin Lin  wrote:
>
> >
> > My personal opinion:
> >
> > If this is a blocking issue, we should tag the issue and raise it in the
> > discussion stage, not in the final release stage, which will waste a lot of
> > time of the release manager. But I see this issue is already closed.
> > https://github.com/apache/pulsar/pull/14097
> >
> > We can evaluate the repair time of this issue. If it does not take too
> > much time, I think it can be merged into 2.9.2. If it is too late, I
> > suggest move it to 2.9.3, since there are other serious issues waiting to
> > be released, we can run in small steps.
> >

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

Il giorno gio 10 feb 2022 alle ore 08:39 PengHui Li
 ha scritto:
>
> Enrico,
>
> There are 40 closed PRs
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.3+is%3Aclosed
> and most of them cherry-picked to branch-2.9, so you mean the 2.9.2

wow, this is incredible, we really should slow down in cherry-picking
stuff to release branches.
But this is off-topic, as this is a VOTE thread.

> contains all of them? or just contain
> https://github.com/apache/pulsar/pull/14192 ?
> I have no objection if we roll out the new RC based on current branch-2.9
> since
> there are some critical fixes for transactions(Of course, I don't think
> there will be any problems moving them to 2.9.3)
> If only for https://github.com/apache/pulsar/pull/14192, I don't think it's
> necessary to roll out the new RC.

I believe it is better to cut a new release candidate out of branch-2.9

Enrico

>
> Thanks,
> Penghui
>
> On Thu, Feb 10, 2022 at 2:28 PM Enrico Olivelli  wrote:
>
> > Gao,
> > The patch https://github.com/apache/pulsar/pull/14192 has been merged.
> > I believe that is better to roll out a new RC and have Pulsar 2.9.2
> > released
> >
> >
> > Thanks for the good discussion
> >
> > Enrico
> >
> > Il giorno gio 10 feb 2022 alle ore 05:11 PengHui Li
> >  ha scritto:
> > >
> > > It should not be a blocking issue, as I mentioned before we can work
> > around
> > > and the issue happens for specific conditions
> > > And there are some other ongoing transaction fixes, for transactions
> > using
> > > the latest branch-2.9 is the best option, not 2.9.2 even contain
> > > https://github.com/apache/pulsar/pull/14097
> > >
> > > So I suggest moving forward with the 2.9.2 release and focusing on the
> > > transaction fixes in 2.9.3.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Thu, Feb 10, 2022 at 11:02 AM Lin Lin  wrote:
> > >
> > > >
> > > > My personal opinion:
> > > >
> > > > If this is a blocking issue, we should tag the issue and raise it in
> > the
> > > > discussion stage, not in the final release stage, which will waste a
> > lot of
> > > > time of the release manager. But I see this issue is already closed.
> > > > https://github.com/apache/pulsar/pull/14097
> > > >
> > > > We can evaluate the repair time of this issue. If it does not take too
> > > > much time, I think it can be merged into 2.9.2. If it is too late, I
> > > > suggest move it to 2.9.3, since there are other serious issues waiting
> > to
> > > > be released, we can run in small steps.
> > > >
> >

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

Il giorno gio 10 feb 2022 alle ore 11:25 PengHui Li
 ha scritto:
>
> > wow, this is incredible, we really should slow down in cherry-picking
> stuff to release branches.
> But this is off-topic, as this is a VOTE thread.
>
> Yes, as discussed here
> https://lists.apache.org/thread/b4wr2chgdrqjgjq4omf6mtfc3g2f9cnx
> 2.9.1 almost equals 2.9.0, 2.9.1 contains the security issue fixes,
> but without any other issue fixes(Includes fixes for some serious issues)
> so commits in 2.9.2 is from 2.9.0 to 2.9.2 lasted more than 2 months.
>
> Most of the PRs cherry-picked to branch-2.9 are bug fixes or important
> enhancements
> such as performance issues. The reason is not we cherry-picked too fast,
> is released too slow. The slower we publish, users have to build by
> themselves,
> if we slow down the cherry-picking, ignore some fixes, this is tantamount
> to pushing users
> to a situation where they maintain their own branches.
> This is definitely not what we want to see.
>
> We should carefully consider whether is it worth delaying the release,
> Explain again, https://github.com/apache/pulsar/pull/14192 is happening in
> specific conditions,
> This often does not occur with the normal use of topics, only for the topic
> will not get a new message.
> The data will not be lost, and we have a way to work around it.l


Please go ahead with the release, I won't VOTE on this thread.
But I hope we can follow up soon with a new release, otherwise due to that bug
you cannot enable transactions on your Pulsar cluster if you have to
support Pulsar client that do not enable transactions

Enrico



>
> Penghui
>
> On Thu, Feb 10, 2022 at 5:03 PM Hang Chen  wrote:
>
> > Hi Enrico,
> >  In my opinion, the transactions is still in developing state and
> > not stable yet. For those critical bug fixes, we'd better prepare
> > 2.9.3 release earlier instead of blocking the current 2.9.2 release.
> > Otherwise, if we make new release candidate for any new find bugs, It
> > will be a huge task for the release manager and make the release
> > process delay too much time.
> >  I suggest to move them to 2.9.3 instead of blocking the 2.9.2 release.
> >
> > Best,
> > Hang
> >
> > Enrico Olivelli  于2022年2月10日周四 16:15写道：
> > >
> > > Il giorno gio 10 feb 2022 alle ore 08:39 PengHui Li
> > >  ha scritto:
> > > >
> > > > Enrico,
> > > >
> > > > There are 40 closed PRs
> > > >
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.3+is%3Aclosed
> > > > and most of them cherry-picked to branch-2.9, so you mean the 2.9.2
> > >
> > > wow, this is incredible, we really should slow down in cherry-picking
> > > stuff to release branches.
> > > But this is off-topic, as this is a VOTE thread.
> > >
> > > > contains all of them? or just contain
> > > > https://github.com/apache/pulsar/pull/14192 ?
> > > > I have no objection if we roll out the new RC based on current
> > branch-2.9
> > > > since
> > > > there are some critical fixes for transactions(Of course, I don't think
> > > > there will be any problems moving them to 2.9.3)
> > > > If only for https://github.com/apache/pulsar/pull/14192, I don't
> > think it's
> > > > necessary to roll out the new RC.
> > >
> > > I believe it is better to cut a new release candidate out of branch-2.9
> > >
> > > Enrico
> > >
> > > >
> > > > Thanks,
> > > > Penghui
> > > >
> > > > On Thu, Feb 10, 2022 at 2:28 PM Enrico Olivelli 
> > wrote:
> > > >
> > > > > Gao,
> > > > > The patch https://github.com/apache/pulsar/pull/14192 has been
> > merged.
> > > > > I believe that is better to roll out a new RC and have Pulsar 2.9.2
> > > > > released
> > > > >
> > > > >
> > > > > Thanks for the good discussion
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno gio 10 feb 2022 alle ore 05:11 PengHui Li
> > > > >  ha scritto:
> > > > > >
> > > > > > It should not be a blocking issue, as I mentioned before we can
> > work
> > > > > around
> > > > > > and the issue happens for specific conditions
> > > > > > And there are some other ongoing transaction fixes, for
> > transactions
> > > > > using
> > > > > >

Re: [ANNOUNCE] New Committer: Aloys Zhang

[Enrico Olivelli]

Congrats!

Enrico

Il giorno gio 10 feb 2022 alle ore 11:55 mattison chao
 ha scritto:
>
> Congratulations,  Aloys Zhang
>
> Best,
> Mattison
>
> > On Feb 10, 2022, at 5:26 PM, Lari Hotari  wrote:
> >
> > Congratulations Aloys Zhang!
> >
> > On Thu, Feb 10, 2022 at 4:46 AM linlin  wrote:
> >
> >> The Apache Pulsar Project Management Committee (PMC) has invited Aloys
> >> Zhang
> >>
> >> (https://github.com/aloyszhang) to become a committer and we are pleased
> >> to
> >>
> >> announce that he has accepted.
> >>
> >> Aloys Zhang joined the Pulsar community in June 2020 and contributed a lot
> >> of commits
> >> to the community, including PIP-70 which brings lightweight broker entry
> >> metadata
> >> to Pulsar and some other pull requests:
> >> https://github.com/apache/pulsar/pulls?q=is%3Apr+assignee%3Aaloyszhang+.
> >>
> >> Welcome and Congratulations, Aloys Zhang! Please enjoy the journey as a
> >> committer :)
> >>
> >> Please join us in congratulating and welcoming Aloys Zhang onboard!
> >>
> >> Best Regards,
> >> Lin Lin on behalf of the Pulsar PMC
> >>
>

New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder

[Enrico Olivelli]

Hello,
I have a use case in which my application MUST use a non-partitioned
topic to work properly, but the topic name is (of course) configurable
by the user.
If I am not using a non-partitioned topic, I do not have string
guarantees on the ordering of the message (because messages will be
spread across multiple partitions).

Currently there is no way to require that the topic IS a
NON-PARTITIONED topic. Especially when the topic does not exist yet,
you configured the namespace to create partitioned topics by default.

Using a PulsarAdmin preliminary call is not a good workaround because
it is expensive, and ideally I want to be able to create the Producer
(or the Consumer) and see that everything works automatically, failing
in case of a partitioned topic.

Thoughts ?


Enrico

Re: [DISCUSS] PIP 141 : Pulsar BOM

[Enrico Olivelli]

Christophe,
It is not clear to me who is the consumer of this BOM.

Java clients should depend on pulsar-client
Pulsar IO and Functions have their own dependencies.

So what's the target? Protocol Handlers? Servlets? Proxy extensions?


Enrico

Il Lun 14 Feb 2022, 15:05 Nicolò Boschi  ha scritto:

> Great idea!
> +1
>
> Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> bornet.ch...@gmail.com> ha scritto:
>
> > Hi everyone,
> >
> > I'd like to open a discussion on PIP 141 : Pulsar BOM
> >
> > The PIP is here : https://github.com/apache/pulsar/issues/14168
> >
> > ## Motivation
> >
> > When designing NAR modules loaded by Nifi in the broker such as protocol
> > handlers, proxy extensions, Pulsar IO connectors, etc..., it's important
> > that the dependencies that are common to the module and the broker are as
> > close as possible to prevent incompatible library exceptions
> > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
> > ...) at runtime. If a class is both in the NAR and in the broker, the
> > broker one will be loaded.
> >
> > ## Goal
> >
> > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > A BOM is a special kind of POM that contains all the dependency versions
> > that are used by the project and can be imported in another project.
> > Currently there is a `dependencyManagement` section in Pulsar's parent
> POM
> > but it's not always possible to derive from this parent POM as it
> imports a
> > lot more things than the dependency versions and external projects
> usually
> > prefer to have their own parent POM.
> > External projects can import this BOM and use the same library versions
> as
> > Pulsar at compile/test time.
> >
> > ## API Changes
> >
> > No API changes
> >
> > ## Implementation
> >
> > The `dependencyManagement` section of Pulsar's parent POM and related
> > properties will be extracted in a POM and put in a `pulsar-bom`
> directory.
> > The `pulsar-bom` artifact shall be built and released independently from
> > the rest of Pulsar project (not a maven module).
> > The Pulsar's parent POM `dependencyManagement` section is replaced by:
> >
> >   
> > 
> >   
> > org.apache.pulsar
> > pulsar-bom
> > 2.10.0-SNAPSHOT
> > pom
> > import
> >   
> > 
> >   
> >
> > The CI will have to build `pulsar-bom` before building Pulsar.
> >
> > ## Reject Alternatives
> >
> > The BOM could be part of a distinct Git project. This would be harder to
> > handle for contributions that modify both the BOM and Pulsar.
> >
> >
> > Thanks a lot for your feedback !
> >
> > Christophe
> >
>
>
> --
> Nicolò Boschi
>

Re: [DISCUSS] PIP 141 : Pulsar BOM

[Enrico Olivelli]

Il giorno lun 14 feb 2022 alle 18:51 Christophe Bornet <
bornet.ch...@gmail.com> ha scritto:

> For plugins that are part of the main project, you get the dependencies
> from the parent POM. But for third-party plugins (including pulsar-io,
> protocol handlers and proxy-extensions),


So the problem is mostly for protocol handlers.

Se have probably a general problem about classpath  isolation,
but probably the major problem is that there is no public API for PHs and
you tend to use internal Pulsar APIs.
Adding new libraries is hard, and also you don’t have control about which
version is loaded, probably even if you load the same version that we in
Pulsar


Enrico


you generally don't want to
> inherit from the Pulsar parent POM and it would be great to be able to
> import a BOM instead.
>
> Christophe
>
> Le lun. 14 févr. 2022 à 16:25, Enrico Olivelli  a
> écrit :
>
> > Christophe,
> > It is not clear to me who is the consumer of this BOM.
> >
> > Java clients should depend on pulsar-client
> > Pulsar IO and Functions have their own dependencies.
> >
> > So what's the target? Protocol Handlers? Servlets? Proxy extensions?
> >
> >
> > Enrico
> >
> > Il Lun 14 Feb 2022, 15:05 Nicolò Boschi  ha
> scritto:
> >
> > > Great idea!
> > > +1
> > >
> > > Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> > > bornet.ch...@gmail.com> ha scritto:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to open a discussion on PIP 141 : Pulsar BOM
> > > >
> > > > The PIP is here : https://github.com/apache/pulsar/issues/14168
> > > >
> > > > ## Motivation
> > > >
> > > > When designing NAR modules loaded by Nifi in the broker such as
> > protocol
> > > > handlers, proxy extensions, Pulsar IO connectors, etc..., it's
> > important
> > > > that the dependencies that are common to the module and the broker
> are
> > as
> > > > close as possible to prevent incompatible library exceptions
> > > > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError,
> etc
> > > > ...) at runtime. If a class is both in the NAR and in the broker, the
> > > > broker one will be loaded.
> > > >
> > > > ## Goal
> > > >
> > > > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > > > A BOM is a special kind of POM that contains all the dependency
> > versions
> > > > that are used by the project and can be imported in another project.
> > > > Currently there is a `dependencyManagement` section in Pulsar's
> parent
> > > POM
> > > > but it's not always possible to derive from this parent POM as it
> > > imports a
> > > > lot more things than the dependency versions and external projects
> > > usually
> > > > prefer to have their own parent POM.
> > > > External projects can import this BOM and use the same library
> versions
> > > as
> > > > Pulsar at compile/test time.
> > > >
> > > > ## API Changes
> > > >
> > > > No API changes
> > > >
> > > > ## Implementation
> > > >
> > > > The `dependencyManagement` section of Pulsar's parent POM and related
> > > > properties will be extracted in a POM and put in a `pulsar-bom`
> > > directory.
> > > > The `pulsar-bom` artifact shall be built and released independently
> > from
> > > > the rest of Pulsar project (not a maven module).
> > > > The Pulsar's parent POM `dependencyManagement` section is replaced
> by:
> > > >
> > > >   
> > > > 
> > > >   
> > > > org.apache.pulsar
> > > > pulsar-bom
> > > > 2.10.0-SNAPSHOT
> > > > pom
> > > > import
> > > >   
> > > > 
> > > >   
> > > >
> > > > The CI will have to build `pulsar-bom` before building Pulsar.
> > > >
> > > > ## Reject Alternatives
> > > >
> > > > The BOM could be part of a distinct Git project. This would be harder
> > to
> > > > handle for contributions that modify both the BOM and Pulsar.
> > > >
> > > >
> > > > Thanks a lot for your feedback !
> > > >
> > > > Christophe
> > > >
> > >
> > >
> > > --
> > > Nicolò Boschi
> > >
> >
>

Re: [ANNOUNCE] New Committer: Li Li

[Enrico Olivelli]

Congratulations

Enrico

Il Mar 15 Feb 2022, 08:07 PengHui Li  ha scritto:

> The Apache Pulsar Project Management Committee (PMC) has invited Li Li
> https://github.com/urfreespace to become a committer and we are pleased to
> announce that he has accepted.
>
> Li Li has done a great contribution to Pulsar Website, documentation.
>
> Welcome and Congratulations, Li Li!
>
> Please join us in congratulating and welcoming Li Li onboard!
>
> Best Regards,
> Penghui Li on behalf of the Pulsar PMC
>
>

Re: Improve quality and efficiency of image creation

[Enrico Olivelli]

Yu,

Il Mar 15 Feb 2022, 13:12 Yu  ha scritto:

> Bumping up this thread, any thoughts? Thanks
>
> On Fri, Jan 28, 2022 at 1:00 AM Yu  wrote:
>
> > Hi Pulsarers,
> >
> >
> > As you may notice, the images in Pulsar documentation are in inconsistent
> > styles, which affect user experience. This issue may be caused by:
> >
> > 1) Pulsar community does not have design style guides.
> >
> > 2) Original images are not shared, so that contributors can not be able
> to
> > re-use the exiting images, and then they create their own.
> >
> >
> > Solving this issue brings many benefits, such as:
> >
> > - Build a trusting relationship with Pulsar users.
> >
> > - Save image contributors' efforts and time in the design process.
> >
> > - Help design work communicates clearly and looks professional.
> >
> >
> > And for 2), are there any good solutions?
> >
> > Hosting original images on GitHub repo may not be a pretty good idea
> > since:
> >
> > - Collaborating synchronously online is impossible. You need to download
> > the original file, re-use/edit it, and upload it back. Few contributors
> > would like to do it.
> >
> > - Occupy much space.
> >
> >
> > Can we choose a professional tool to create/edit/host images and
> > collaborate online synchronously?
> >
> > Just my two cents: for example, can ASF give the community an email
> > address and register an account on LucidChart? PMC can manage the
> > LucidChart Pulsar account and all image assets and grant different access
> > to contributors.


How much does it cost?

Enrico

So that contributors can re-use/collaborate in an
> > efficient manner.
> >
> >
> > Please correct me if my understanding is incorrect, I'd love your
> > feedback, thanks!
> >
>

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

[Enrico Olivelli]

Ran,
Are we cutting the new RC from the current branch-2.9?

Enrico

Il Mar 15 Feb 2022, 14:57 PengHui Li  ha scritto:

> Hi Ran,
>
> I think all the PRs that block the 2.9.2 release are merged.
> Could you please help cherry-pick the PRs and start a new RC?
>
> Thanks,
> Penghui
>
> On Mon, Feb 14, 2022 at 8:25 PM PengHui Li  wrote:
>
> > Hi Lari,
> >
> > We have another issue that needs to confirm if it will introduce break
> > changes in 2.9.2,
> > Expected to have a result tomorrow, it related to
> > https://github.com/apache/pulsar/pull/13383,
> > We're doing more testing to make sure it doesn't introduce unexpected
> > behavior.
> >
> > Regards,
> > Penghui
> >
> > On Mon, Feb 14, 2022 at 8:10 PM Lari Hotari  wrote:
> >
> >> When is 2.9.2 Candidate 3 planned?
> >> What changes will it include? All current changes in branch-2.9 ?
> >> The version has already been set to 2.9.3-SNAPSHOT in branch-2.9 with
> >> https://github.com/apache/pulsar/pull/14089 . If we do 2.9.2 with all
> >> current changes from branch-2.9, the commit for PR 14089 would have to
> be
> >> reverted before the next release.
> >> Another possibility is to skip 2.9.2 completely and proceed directly
> with
> >> 2.9.3 release.
> >>
> >> -Lari
> >>
> >> On 2022/02/11 08:28:58 PengHui Li wrote:
> >> > Now, there is a regression introduced in 2.9.2
> >> >
> >> > I have pushed out the fix https://github.com/apache/pulsar/pull/14231
> ,
> >> PTAL.
> >> >
> >> > -1 from my side
> >> >
> >> > Need to get the fix merged and roll out the new RC3 @Ran
> >> >
> >> > Regards,
> >> > Penghui
> >> >
> >> > On Thu, Feb 10, 2022 at 9:54 PM Nicolò Boschi 
> >> wrote:
> >> >
> >> > > Penghui,
> >> > >
> >> > >
> >> > > I didn't know that there were so many known bugs around transactions
> >> > > scheduled for 2.9.3, my bad.
> >> > >
> >> > > However, as Enrico pointed out, the issue impacts Pulsar clients
> that
> >> are
> >> > > not using the transactions, so we can't just say - ok, just another
> >> bug
> >> > > about transactions, it's not critical since they're not production
> >> ready
> >> > > (btw, where we state that they aren't production ready on the
> >> > > documentation?).
> >> > >
> >> > >
> >> > > The workaround you mentioned is not always viable, since you can
> have
> >> > > clients of different tenants/customers that are not using
> transactions
> >> > > while, at the same time, a little portion that are experiencing with
> >> them.
> >> > >
> >> > > I agree that it is uncommon to have only one message produced. On
> the
> >> other
> >> > > hand, it's a very common case where other projects using Pulsar have
> >> > > unit/integration tests that write only one message and expect to be
> >> > > consumed (that's because they test the application logic and not
> >> Pulsar).
> >> > >
> >> > >
> >> > > Given that, it's fair to say that 2.9.2 is not worse than 2.9.1, so,
> >> > > finally, we can go ahead.
> >> > >
> >> > > Looking forward to see 2.9.3 soon
> >> > >
> >> > >
> >> > > I tested the artifacts, so I'll put my vote here:
> >> > >
> >> > >
> >> > > +1 (non binding)
> >> > >
> >> > >
> >> > > Checks:
> >> > >
> >> > > - Checksum and signatures
> >> > >
> >> > > - Apache Rat check passes
> >> > >
> >> > > - Compile from source w JDK11
> >> > >
> >> > > - Build docker image from source
> >> > >
> >> > > - Run Pulsar standalone and produce-consume from CLI
> >> > >
> >> > >
> >> > > BR,
> >> > >
> >> > > Nicolò
> >> > >
> >> > > Il giorno gio 10 feb 2022 alle ore 13:39 PengHui Li <
> >> peng...@apache.org>
> >> > > ha
> >> > > scritto:
> >> > >
> >> > > > > Please go ahead with the release, I won't VOTE on this thread.
> >> > > > But I hope we can follow up soon with a new release, otherwise due
> >> to
> >> > > that
> >> > > > bug
> >> > > > you cannot enable transactions on your Pulsar cluster if you have
> to
> >> > > > support Pulsar client that do not enable transactions
> >> > > >
> >> > > >
> >> > > > Yes, agree. We will follow up the 2.9.3 soon. There are other
> >> > > > ongoing transaction fixes
> >> > > > we will complete them ASAP and provide a version with certain
> >> guarantees
> >> > > > for transaction stability.
> >> > > > We are doing lots of tests these days, 2.9.3 should be a good
> >> version for
> >> > > > transactions.
> >> > > >
> >> > > > Thanks,
> >> > > > Penghui
> >> > > >
> >> > > >
> >> > > > On Thu, Feb 10, 2022 at 7:37 PM Lin Lin 
> wrote:
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > +1(binding)
> >> > > > >
> >> > > > > 1. Checked the signature
> >> > > > > 2. Start standalone
> >> > > > > 3. Publish and consume successfully
> >> > > > > 4. Checked function
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Nicolò Boschi
> >> > >
> >> >
> >>
> >
>