Re: Status of Pulsar 2.9.0 and starting 2.9.1

2021-12-10 Thread Lari Hotari
+1 la 11. jouluk. 2021 klo 9.07 Enrico Olivelli kirjoitti: > Hello folks, > Yesterday we committed the release notes for 2.9.0. > I just have to publish a couple of other artifacts and update the website > before announcing 2.9.0. > My plan is to complete the procedure next week. > > In the mean

Re: [Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions

2021-12-11 Thread Lari Hotari
contains the fix). BR, Lari On Fri, Dec 10, 2021 at 2:23 PM Lari Hotari wrote: > > As many of you might have already heard of this, there's a severe RCE > 0-day exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1). > Blog post: https://www.lunasec.io/docs/blog/log

Re: [ANNOUNCE] New Committer: Marvin Cai

2021-12-13 Thread Lari Hotari
Congrats Marvin, well deserved! -Lari On Mon, Dec 13, 2021 at 11:46 AM linlin wrote: > The Apache Pulsar Project Management Committee (PMC) has invited Marvin Cai > https://github.com/MarvinCai to become a committer and we are pleased to > announce that he has accepted. > > Marvin has joined th

Analysis of impact of most recent Log4j CVE, CVE-2021-45046 to Pulsar

2021-12-15 Thread Lari Hotari
There's a new CVE, CVE-2021-45046 in Log4j < 2.16.0 details: https://blogs.apache.org/foundation/entry/apache-log4j-cves Summary: Pulsar isn't impacted with CVE-2021-44228 when the default log4j configuration is used. However, remember that Pulsar is impacted by the actual Log4Shell CVE and Pulsa

Re: [VOTE] Apache Pulsar 2.8.2 candidate 1

2021-12-15 Thread Lari Hotari
-1, I verified the packages, but the signatures are invalid. It looks like the https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/apache-pulsar-2.8.2-bin.tar.gz.asc file is from December 1st. I checked this in the SVN staging repository history ( https://dist.apache.org/repos

Re: Dropping Presto SQL in 2.9.0 - status ?

2021-12-16 Thread Lari Hotari
t still want > to keep a copy of the connector in Pulsar repo and push changes to Trino > repo periodically, as this will allow much faster bug fix and feature > iteration. > Best, > Marvin, > > Lari Hotari 于2021年11月17日周三 下午2:19写道: > > > Dear Pulsar community members

Yet another CVE-2021-45105 in Log4j < 2.17.0, analysis of impact to Pulsar

2021-12-18 Thread Lari Hotari
nstance/JavaInstanceRunnable.java#L179-L182 Pulsar name validation pattern: https://github.com/apache/pulsar/blob/7bf14b5ac049d71c7ff74bbe758cb41aaffeb0af/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java#L31-L34 Regards, Lari Hotari

[DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

2021-12-21 Thread Lari Hotari
Dear Pulsar community members, PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate repositories" was created in April 2020. The repositories for pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year ago [2]. I'd like to propose that we continue with t

Re: OWASP dependencies check on active branches

2021-12-22 Thread Lari Hotari
Good work Nicolò! It's great to have OWASP dependency check handled for all active branches. -Lari On Wed, Dec 22, 2021 at 5:05 PM Nicolò Boschi wrote: > Hello everyone, > > I created a couple of pull requests in order to run a periodic check on > Pulsar active branches. In this way we can proa

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

2021-12-23 Thread Lari Hotari
ie > > On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli > wrote: > > > Lari, > > > > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari > > > ha scritto: > > > > > Dear Pulsar community members, > > > > > > PIP-62[1], "PIP 62:

upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-03 Thread Lari Hotari
Hi all, There's an upcoming change in the Apache Pulsar Helm chart to finally switch to Pulsar 2.8.x, more specifically to Apache Pulsar version 2.8.2 . The latest Apache Pulsar Helm Chart release uses the Apache Pulsar 2.7.4 image. The pull request to switch to Apache Pulsar image version 2.8.2

Lifting Kubernetes minimum version requirement for Apache Pulsar Helm Charts from k8s 1.14 to 1.18

2022-01-04 Thread Lari Hotari
Hi all, Currently k8s 1.14 version is used in CI to verify the Helm chart changes. k8s 1.14 became end-of-life 2019-12-11 , over 2 years ago [1]. The oldest maintained version for Kubernetes is 1.20 and it will become end-of-life on 2022-02-28, in less than 2 months from now [2]. There's a PR to

Re: Lifting Kubernetes minimum version requirement for Apache Pulsar Helm Charts from k8s 1.14 to 1.18

2022-01-05 Thread Lari Hotari
Reminder, there's a PR to lift the minimum requirement for Apache Pulsar Helm Charts to 1.18 so that we don't fall too far behind. https://github.com/apache/pulsar-helm-chart/pull/192 Please review BR, Lari On Tue, Jan 4, 2022 at 11:40 AM Lari Hotari wrote: > Hi all, > >

Re: [DISCUSS] Apache Pulsar 2.9.2 release

2022-01-05 Thread Lari Hotari
+1 On Wed, Jan 5, 2022 at 11:23 AM Ran Gao wrote: > Hello, Pulsar community: > > I'd like to propose that we release Apache Pulsar 2.9.2. > > Currently, compared to 2.9.1, branch-2.9 already merged 171 commits(refer > to [0]), they contain the log4j security patch and many important fixes. > > I

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-05 Thread Lari Hotari
't support TLS and the impacted NettyServerCnxnFactory must be used for TLS. How do we handle the decision? Can we proceed in merging https://github.com/apache/pulsar-helm-chart/pull/190 regardless of the known issue? BR, Lari On Mon, Jan 3, 2022 at 4:39 PM Lari Hotari wrote: > Hi a

Re: Lifting Kubernetes minimum version requirement for Apache Pulsar Helm Charts from k8s 1.14 to 1.18

2022-01-07 Thread Lari Hotari
? I see most of the changes are done > in the CI part. > > - Sijie > > On Tue, Jan 4, 2022 at 1:41 AM Lari Hotari wrote: > > > Hi all, > > > > Currently k8s 1.14 version is used in CI to verify the Helm chart > changes. > > > > k8s 1.14 became

Re: Lifting Kubernetes minimum version requirement for Apache Pulsar Helm Charts from k8s 1.14 to 1.18

2022-01-09 Thread Lari Hotari
atest released versions. BR, Lari On Tue, Jan 4, 2022 at 11:40 AM Lari Hotari wrote: > Hi all, > > Currently k8s 1.14 version is used in CI to verify the Helm chart changes. > > k8s 1.14 became end-of-life 2019-12-11 , over 2 years ago [1]. > The oldest maintained version for Ku

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-12 Thread Lari Hotari
utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > 不含病毒。www.avg.com > > < > > > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > >

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-26 Thread Lari Hotari
art/pull/190 asap so that we can finally move to Pulsar 2.8.x in the Pulsar Helm Chart. BR, Lari On 2022/01/12 12:52:47 Lari Hotari wrote: > Hi Sijie, > > Thanks for the suggestions. > > > That means: > > > > > 1. We should have a separate `version` from `appVers

Re: upcoming change: Apache Pulsar Helm Chart switching from Pulsar 2.7.4 version to 2.8.2; known issue with ZK when TLS is enabled

2022-01-26 Thread Lari Hotari
rk Lari ! > > what about upgrading to 2.9.1 and not to 2.8.2 ? > We are VOTing for 2.9.2 and 2.10 will be shipped soon > > isn't 2.8.2 quite old at this point ? > > Enrico > > Il giorno mer 26 gen 2022 alle ore 14:52 Lari Hotari > ha scritto: > >

Pulsar Flaky test report 2022-02-03 for PR builds in CI

2022-02-03 Thread Lari Hotari
Dear Pulsar community members, Here's a report of the flaky tests in Pulsar CI during the observation period of 2022-01-27 to 2022-02-03 . The full report is available as a Google Sheet, https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ There are a lot more flaky

Re: Pulsar Flaky test report 2022-02-03 for PR builds in CI

2022-02-03 Thread Lari Hotari
I added the links to GitHub issues to the spreadsheet: https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit#gid=456314619 Let's focus fixing the top 10 most flaky tests asap. Please comment on the issue that you are working on it so that we don't unnecessarily do

Re: [DISCUSS] Release Pulsar 2.8.3

2022-02-09 Thread Lari Hotari
+1 Thank you, Michael, for volunteering to be the release manager for 2.8.3. -Lari On Wed, Feb 9, 2022 at 8:16 PM Michael Marshall wrote: > Hello Pulsar Community, > > We have had several important fixes since we released 2.8.2 a month > ago. I propose we start the process to release 2.8.3, and

Re: [DISCUSS] Releasing Pulsar-client-go 0.8.0

2022-02-09 Thread Lari Hotari
+1 -Lari to 10. helmik. 2022 klo 4.44 r...@apache.org kirjoitti: > Hello Everyone: > > > I hope you’ve all been doing well. In the past two months, we have > > fixed a number of bugs related to connection leaks and added > > some new features. For more information refer to: > > > https://github

[DISCUSS] Release Pulsar 2.7.5

2022-02-09 Thread Lari Hotari
Hello Pulsar Community, We have had several important fixes since we released 2.7.4 on Dec 27, 2021. I propose we start the process to release 2.7.5, and I volunteer to be the release manager. Here [0] you can find the list of 11 commits to branch-2.7 since the 2.7.4 release. There are 3 closed P

Re: [ANNOUNCE] New Committer: Aloys Zhang

2022-02-10 Thread Lari Hotari
Congratulations Aloys Zhang! On Thu, Feb 10, 2022 at 4:46 AM linlin wrote: > The Apache Pulsar Project Management Committee (PMC) has invited Aloys > Zhang > > (https://github.com/aloyszhang) to become a committer and we are pleased > to > > announce that he has accepted. > > Aloys Zhang joined

[VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-11 Thread Lari Hotari
This is the first release candidate for Apache Pulsar, version 2.7.5. It contains the following commits after the previous release: https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-1 *** Please download, test and vote on this release. This vote will stay open for at least 72 hou

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-14 Thread Lari Hotari
Please help to verify this release. Thank you -Lari On 2022/02/11 13:47:50 Lari Hotari wrote: > This is the first release candidate for Apache Pulsar, version 2.7.5. > > It contains the following commits after the previous release: > https://github.com/apache/pulsar/compare/v2.

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-14 Thread Lari Hotari
> After the features are completed, I will create the new 2.10 branch, and > only apply > the critical bug fixes, regression fixes. So that we can have adequate > testing on branch-2.10 Hi Penghui, What's the status of 2.10.0 release? What features aren't complete? In PIP 47 (https://github.co

Re: [VOTE] Pulsar Release 2.9.2 Candidate 2

2022-02-14 Thread Lari Hotari
When is 2.9.2 Candidate 3 planned? What changes will it include? All current changes in branch-2.9 ? The version has already been set to 2.9.3-SNAPSHOT in branch-2.9 with https://github.com/apache/pulsar/pull/14089 . If we do 2.9.2 with all current changes from branch-2.9, the commit for PR 14089

Re: [DISCUSS] PIP 141 : Pulsar BOM

2022-02-14 Thread Lari Hotari
> It is not clear to me who is the consumer of this BOM. I was thinking that using a Maven BOM file [1] could be useful when implementing "PIP 62: Move connectors, adapters and Pulsar Presto to separate repositories" [2]. Pulsar Adapters is the only part of PIP-62 that has been moved. The duplicat

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-15 Thread Lari Hotari
Thanks for the detailed reply, Penghui. > And, for the new metadata API, we found an issue that will introduce the > cache inconsistent issue, > we are working on a fix, it should be a release blocker, otherwise, > 2.10 will not able to use. Was this about the issue which this PR https://github.

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-15 Thread Lari Hotari
+1 > > > > Checks: > > > > - Checksum and signatures > > > > - Compile from source w JDK11 > > > > - Validate Pub/Sub and Java Functions [1] > > > > - Validate Connectors [1] > > > > - Validate Stateful Functions [1] > >

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-15 Thread Lari Hotari
14240 does not run > any tests. > All the tests were completed in 1 min, please check. > > Thanks, > Penghui > > > > On Tue, Feb 15, 2022 at 7:23 PM Lari Hotari wrote: > > > The tests passed for the a PR based on revision a27e0853bda in branch-2.7 >

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-15 Thread Lari Hotari
On 2022/02/15 14:13:59 PengHui Li wrote: > The rationale for these changes, I think it starts from this PR > https://github.com/apache/pulsar/pull/13666 > This is the only one example, we have seen the same issue again and again. > After #13666 get merged, > The contributors found there are many pl

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-16 Thread Lari Hotari
+1 On 2022/02/11 13:47:50 Lari Hotari wrote: > This is the first release candidate for Apache Pulsar, version 2.7.5. > > It contains the following commits after the previous release: > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-1 > > *** Please download

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-16 Thread Lari Hotari
> The PR you shared https://github.com/apache/pulsar/pull/14240 does not run > any tests. > All the tests were completed in 1 min, please check. > > Thanks, > Penghui > > > > On Tue, Feb 15, 2022 at 7:23 PM Lari Hotari wrote: > > > The tests passed for

Re: [VOTE] Pulsar Release 2.7.5 Candidate 1

2022-02-16 Thread Lari Hotari
I'll vote again with -1. The "CI - Integration - Cli / cli" build job passes cleanly for 2.7.4 . This means that it's very likely that there's a regression in 2.7.5 . I'll investigate the issue. -Lari On 2022/02/16 10:21:23 Lari Hotari wrote: > The branch-2.7

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-16 Thread Lari Hotari
n be achieved if waiting is done asynchronously." You can achieve better resource efficiency and better scalability with asynchronous servlets. Do we have such problems in Apache Pulsar with the Servlet API? -Lari On 2022/02/15 14:51:59 Lari Hotari wrote: > On 2022/02/15 14:13:59 PengH

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-16 Thread Lari Hotari
On 2022/02/16 00:58:20 PengHui Li wrote: > Which is a sync method. Ultimately this could lead to all the pulsar-web > thread > blocked. we'd better not introduce blocking calls if we use AsyncResponse. > > > What issue did you see? Please share more context. Thanks for the > patience. > > It happ

Re: [DISCUSS] Apache Pulsar 2.10.0 release

2022-02-16 Thread Lari Hotari
I created PR https://github.com/apache/pulsar/pull/14320 to set numHttpServerThreads=200 . Please review On 2022/02/16 12:39:34 Lari Hotari wrote: > On 2022/02/16 00:58:20 PengHui Li wrote: > > Which is a sync method. Ultimately this could lead to all the pulsar-web > > thread &g

[DISCUSS] PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability

2022-02-16 Thread Lari Hotari
URL: https://github.com/apache/pulsar/issues/14329 Motivation Since Pulsar Admin API uses the blocking servlet API, all Jetty threads might be occupied and this causes unavailability of the Pulsar Admin API. The default value for the maximum number of threads for Jetty is too low in Pulsar. That

Re: Pulsar Helm Chart Repo is broken

2022-02-16 Thread Lari Hotari
The chart index is now published, https://pulsar.apache.org/charts/index.yaml . However, the history isn't there. I'll fix that with a manual commit. -Lari On Thu, Feb 17, 2022 at 7:31 AM Guangning E wrote: > yes, this pr for fixed it => > https://github.com/apache/pulsar-helm-chart/pull/231 >

Re: Pulsar Helm Chart Repo is broken

2022-02-16 Thread Lari Hotari
://pulsar.apache.org/charts/index.yaml?dummy . GitHub pages caching is explained in this SO answer: https://stackoverflow.com/questions/24851824/how-long-does-it-take-for-github-page-to-show-changes-after-changing-index-html . -Lari On Thu, Feb 17, 2022 at 7:52 AM Lari Hotari wrote: > The chart in

Re: Rollback Pulsar Website Change

2022-02-17 Thread Lari Hotari
Change to roll back Apache Pulsar Helm chart publishing to new site: https://github.com/apache/pulsar-helm-chart/pull/234 -Lari On 2022/02/17 19:44:53 Sijie Guo wrote: > https://github.com/apache/pulsar-site/pull/3 > https://github.com/apache/pulsar/pull/14356 > > Changes to roll back the websit

Re: [VOTE] Pulsar Release 2.10.0 Candidate 2

2022-03-02 Thread Lari Hotari
-1 There was a thread leak in MLTransactionMetadataStore which also impacts production code. It is fixed by https://github.com/apache/pulsar/pull/14524 which is now merged to master branch. The thread leak resulted in a memory leak in tests and that's how I noticed it. The thread leak was also

Re: [DISUSS] Improve unit test stability

2022-03-08 Thread Lari Hotari
I think this is a good idea. In CI, the tests will be retried multiple times at multiple levels: at the test class / test method level and at the build level. I'd like to get rid of test retries completely, since that is the reason why the flakiness problem gets worse over time. This was also cove

Re: [DISUSS] Improve unit test stability

2022-03-08 Thread Lari Hotari
Here's a PR to disable tests retries for new and modified tests: https://github.com/apache/pulsar/pull/14607 Please review and provide feedback. -Lari On 2022/03/08 11:24:13 Lari Hotari wrote: > I think this is a good idea. In CI, the tests will be retried multiple > times at mult

Re: [DISUSS] Improve unit test stability

2022-03-09 Thread Lari Hotari
; I see. Now we retry all tests by default. What's the reason? > >> > >> Test retries are enabled precisely because some tests are flaky. > >> > >> Thanks, > >> Michael > >> > >> > >> On Tue, Mar 8, 2022 at 7

Re: [DISUSS] Improve unit test stability

2022-03-09 Thread Lari Hotari
> > Good job! > > > > I see. Now we retry all tests by default. What's the reason? > > > > The process I understand is: > > We don't need to retry the unit test. Once the test fails, let the job fail. > > The difference is that new or modified te

Re: [ANNOUNCE] New Committer: Andrey Yegorov

2022-03-09 Thread Lari Hotari
Congrats Andrey! -Lari On 2022/03/07 19:25:44 Dave Fisher wrote: > The Apache Pulsar Project Management Committee (PMC) has invited Andrey > Yegorov > https://github.com/dlg99 to become a committer and we are pleased to > announce that he has accepted. > > Andrey has made great contributions to

Re: [DISCUSS] Migrate TestNg Assertion to AssertJ

2022-03-16 Thread Lari Hotari
I support the switch to AssertJ. Some automated tooling might be useful for doing the bulk of the migration. AssertJ provides some scripts for the migration: https://joel-costigliola.github.io/assertj/assertj-core-converting-testng-assertions-to-assertj.html There's also more sophisticated tools

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-22 Thread Lari Hotari
n merge > them into one workflow. > > - Sijie > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari wrote: > > > Thanks for the feedback Sijie. > > > > > We are using a lazy consensus approach. Typically if there is no > > objection, > > >

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-23 Thread Lari Hotari
I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 . BR, -Lari On 2022/03/22 13:38:36 Enrico Olivelli wrote: > Lari, > > Il Mar 22 Mar 2022, 14:32 Lari Hotari ha scritto: > > > I ha

Re: Broker freeze for communications in v 2.7.4

2022-03-23 Thread Lari Hotari
Hi, Thank you for the problem report. Have you already filed an issue in https://github.com/apache/pulsar/issues ? If not, I think it will be helpful for tracking the issue. When there are such issues where the broker seems to freeze, it is helpful to get a threaddump from the frozen broker. T

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-23 Thread Lari Hotari
I have created a separate PR to make pulsarbot support the new GitHub Actions feature of rerunning failed jobs (instead of all jobs in a workflow). Please review https://github.com/apache/pulsar-test-infra/pull/27 too. -Lari On 2022/03/23 10:46:10 Lari Hotari wrote: > I have submitted the

Re: [Guideline] Pulsar PR Naming Convention

2022-03-24 Thread Lari Hotari
The problem with the "Pulsar PR Naming Convention" is that it can conflict with general Git message conventions. The PR title automatically becomes the Git commit's subject line. That's how the PR title related to the Git commit's subject line (title). In Git, the convention is to have a subje

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-28 Thread Lari Hotari
oring workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts On 2022/03/23 10:46:10 Lari Hotari wrote: > I have submitted the PR for refactoring the apache/pulsar GitHub Actions > based CI. Please review https://github.com/apache/pulsar

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-29 Thread Lari Hotari
The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow. Please provide feedback now if you want to do that before the PR is merged. Thanks! -Lari On 2022/03/28 20:05:14 Lari Hotari wrote: > The PR https://github.com/apache/pulsar/pull/14819 is now ready fo

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-29 Thread Lari Hotari
hat might pop up. > > Thanks, > Michael > > On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari wrote: > > > > The PR has sufficient reviews, and I'll proceed with merging it today or > > tomorrow. > > Please provide feedback now if you want to do that befor

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-30 Thread Lari Hotari
:45) at io.prestosql.server.PrestoServer.run(PrestoServer.java:78) at io.prestosql.$gen.Presto_33220220330_100314_1.run(Unknown Source) at io.prestosql.server.PrestoServer.main(PrestoServer.java:72) I'll apply a workaround to unblock CI. -Lari On 2022/03/30 06:52:38 Lari Hotari wrote: > Th

Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"

2022-03-30 Thread Lari Hotari
/reopen it to pick up changes for the new Pulsar CI workflow. That is necessary so that PRs can be merged. -Lari On 2022/03/30 14:08:17 Lari Hotari wrote: > > Merging the PR is blocked by https://github.com/apache/pulsar/issues/14951 . > > Pulsar SQL doesn't work with Java

Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released

2022-03-30 Thread Lari Hotari
QL. I would assume that it's fine for doing this in the master branch. Are there any ideas how to resolve https://github.com/apache/pulsar/issues/14951 in some other way? BR, Lari On 2021/12/23 08:42:57 Lari Hotari wrote: > > I believe we want to keep SQL until the code chang

Please close&reopen your PR (or rebase it) to pick up Pulsar CI fixes

2022-03-31 Thread Lari Hotari
Hi all, The Pulsar IO integration tests were constantly failing and blocking PR jobs from being merged. This issue has been fixed by reverting the Rocksdb upgrade. It turns out that Rocksdb broke binary compatibility in 6.17.3 and code compiled for <6.17.3 isn't fully compatible at runtime wit

Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-01 Thread Lari Hotari
Hi all, There's a small limitation in re-running failed jobs (builds that fail because of flaky tests) in the refactored Pulsar CI workflow which combines multiple jobs into a single workflow. The limitation is that you need to wait for all jobs to complete before failed jobs can be re-run. Ye

Re: Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-01 Thread Lari Hotari
ck up changes from master. Don't close & reopen PRs as I had advised earlier since it causes problems. The wrong builds will be run and that adds up in the build queue. -Lari On 2022/04/01 08:38:54 Lari Hotari wrote: > Hi all, > > There's a small limitation in re-runnin

Re: [discuss] Support fail-fast strategy when broker rate-limited

2022-04-07 Thread Lari Hotari
> IMO, the more serious problem about `disableAutoread` is > Producer/Consumer isolation issue, > since they shares the connection. For example, if topic-A is rate-limited, > topic-B in the same client is also affected. Exactly. The rate-limiting won't even work properly since other rate limiters

Re: Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-08 Thread Lari Hotari
ve the issues together. I'll try to find a place to document the details that are mentioned in this email thread. -Lari On 2022/04/01 14:34:02 Lari Hotari wrote: > I now realized that my advice to close & reopen PRs to pick up master branch > changes is problematic. This will cause

Re: Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-14 Thread Lari Hotari
oesn't work when it cannot find the failed or cancelled workflow runs. -Lari On 2022/04/08 07:01:33 Lari Hotari wrote: > With the new GitHub Actions CI workflow there are cases where you see a red > mark as a failure, but there's no need to rerun failed jobs since the red > fa

Re: PIP-156: Build and Run Pulsar Server on Java 17

2022-04-20 Thread Lari Hotari
Hi Heesung, I support this proposal, but I would have expected you to first reply on the email thread started by Nicolo in February https://lists.apache.org/thread/c0k8p9vy5wyp9l70mt980gdy10smx6qb . Nicolo Boschi has been working on Java 17 compatibility for months and also brought this up to

Re: Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-21 Thread Lari Hotari
be able to rerun the failed jobs. -Lari On 2022/04/01 14:34:02 Lari Hotari wrote: > I now realized that my advice to close & reopen PRs to pick up master branch > changes is problematic. This will cause issues with "/pulsarbot > rerun-failure-checks". The script current

Re: Re-running failed flaky builds in refactored Pulsar CI GitHub Actions workflow

2022-04-22 Thread Lari Hotari
k now. I'm sorry for the inconvenience that it caused when it wasn't working for all cases. Please let me know if there are any remaining issues. -Lari On 2022/04/21 09:45:37 Lari Hotari wrote: > I have made a fix to the problem described below. > Please review https://github.co

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-02 Thread Lari Hotari
-1. It's too early to start a vote. Let's first have discussions. -Lari ma 2. toukok. 2022 klo 9.50 Andras Beni kirjoitti: > Hi Pulsar Community, > > I would like to start a VOTE on "Bucketing topic metadata to allow more > topics per namespace" (PIP-157). > > The proposal can be read at https:

Re: [ANNOUNCE] new Committer Qiang Zhao (mattisonchao)

2022-05-04 Thread Lari Hotari
Congrats Qiang! -Lari ke 4. toukok. 2022 klo 9.28 Enrico Olivelli kirjoitti: > The Project Management Committee (PMC) for Apache Pulsar > has invited Qiang Zhao to become a committer and we are pleased > to announce that he has accepted. > > Qiang Zhao (with GH id mattisonchao) contributed lot

Re: [VOTE] PIP-156: Build and Run Pulsar Server on Java 17

2022-05-09 Thread Lari Hotari
PIP-156 PR https://github.com/apache/pulsar/pull/15264 has been merged to master branch. Please notice that Java 17 is now required for building Pulsar master branch. btw. https://sdkman.io/ is handy for managing multiple JDK versions in local development environments. -Lari On 2022/04/20 16

Re: [VOTE] PIP-161 Exclusive Producer: new mode ExclusiveWithFencing

2022-05-17 Thread Lari Hotari
+1 -Lari On 2022/05/16 07:09:19 Enrico Olivelli wrote: > This is the VOTE thread for PIP-161 > > This is the GH issue: https://github.com/apache/pulsar/issues/15528 > This is the PR: https://github.com/apache/pulsar/pull/15488 > > Motivation > > In PIP-68 we introduced two access modes for the

[DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface

2022-05-20 Thread Lari Hotari
Hi all, I started writing this email as feedback to "PIP-157: Bucketing topic metadata to allow more topics per namespace" [3]. This email expanded to cover some analysis of "PIP-45: Pluggable metadata interface" [4] design. (A good introduction to PIP-45 is the StreamNative blog post "Moving Towa

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-20 Thread Lari Hotari
t affected at all by this > proposal, as the naming happens on top of it). > > Some parts can be clarified (as it is always the case), though I don't > think it's a good idea to stop a vote at this point. > > > Matteo > > -- > Matteo Merli > >

Pulsar Flaky test report 2022-06-02 for PR builds in CI

2022-06-02 Thread Lari Hotari
Dear Pulsar community members, Here's a report of the flaky tests in Pulsar CI during the observation period of 2022-05-26 to 2022-06-02 . The full report is available as a Google Sheet, https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing The repor

Re: [DISCUSS] PIP-174: Provide new implementation for broker dispatch cache

2022-06-07 Thread Lari Hotari
This is a very useful proposal. LGTM -Lari On Tue, Jun 7, 2022 at 3:48 AM Matteo Merli wrote: > https://github.com/apache/pulsar/issues/15954 > > WIP can be seen at: https://github.com/apache/pulsar/pull/15955 > > --- > > > ## Motivation > > The current implementation of the read cache

Re: [DISCUSS] Move Pulsar SQL to a separated repository?

2022-07-18 Thread Lari Hotari
Thanks for picking up this task. The decision to move Pulsar SQL out of apache/pulsar repository has been made over 2 years ago in April 2020 with PIP-62, https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories . It's not only abou

Re: [Vote] PIP 198: Standardize PR Naming Convention using GitHub Actions

2022-08-09 Thread Lari Hotari
+1, with some conditions about the details of PIP 198 that are listed below: Would it be possible to improve the proposal in a way that the valid prefixes for type and component are in a file in the repository and the possible checker would use this file as the source of truth? Tison already poi

Re: [DISCUSS] Create a new Github Project to track the flaky tests

2022-08-12 Thread Lari Hotari
Thank you Penghui, really useful way to coordinate the fixing of flaky tests! -Lari On Wed, Aug 10, 2022 at 11:35 AM PengHui Li wrote: > Hi all, > > For better tracking flaky test fix, I have tried to create a Github > Project under the Pulsar repo https://github.com/apache/pulsar/projects/11 >

Re: [DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface

2022-08-15 Thread Lari Hotari
Bumping up this thread. -Lari pe 20. toukok. 2022 klo 1.57 Lari Hotari kirjoitti: > Hi all, > > I started writing this email as feedback to "PIP-157: Bucketing topic > metadata to allow more topics per namespace" [3]. > This email expanded to cover some analysis of &q

Re: [VOTE] PIP-193 : Sink preprocessing Function

2022-08-16 Thread Lari Hotari
+1 (binding) Good proposal. My only comment is about the naming. Instead of calling it a "preprocessing function" or "extra function", I'd propose calling it "decorator function". -Lari On 2022/07/28 10:39:35 Christophe Bornet wrote: > Hi, Pulsar community, > > I'd like to start a vote on PIP

Performance issue in Pulsar with large fan-outs since Pulsar 2.8.2

2022-08-21 Thread Lari Hotari
Hi all, I'd like to get some more eyes on this long outstanding performance issue with large fan-outs (a large number of consumers for a single topic). The broker cache does not work as expected due to invalid changes introduced in version 2.8.2 by PR https://github.com/apache/pulsar/pull/12045.

Re: Performance issue in Pulsar with large fan-outs since Pulsar 2.8.2

2022-08-25 Thread Lari Hotari
naged-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainer.java#L109 > [1] https://github.com/apache/pulsar/pull/12045 > [2] https://github.com/apache/pulsar/pull/14985 > > > On Sun, Aug 21, 2022 at 5:47 PM Lari Hotari wrote: > > > > Hi all, &

Pulsar CI congested, master branch build broken

2022-08-26 Thread Lari Hotari
Hi, GitHub Actions builds have been piling up in the build queue in the last few days. I posted on bui...@apache.org https://lists.apache.org/thread/6lbqr0f6mqt9s8ggollp5kj2nv7rlo9s and created INFRA ticket https://issues.apache.org/jira/browse/INFRA-23633 about this issue. There's also a threa

Re: [DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface

2022-08-29 Thread Lari Hotari
or project direction are disallowed; off-list discourse and transactions must be brought on-list." On Tue, Aug 16, 2022 at 6:15 AM Lari Hotari wrote: > Bumping up this thread. > > -Lari > > pe 20. toukok. 2022 klo 1.57 Lari Hotari kirjoitti: > >> Hi all, >> &

Re: [DISCUSS] Introduce FlowControl to metrics endpoint

2022-08-29 Thread Lari Hotari
Good reiteration of the problem and good points, Asaf. I'd like to add a new aspect to the proposal: there might be other solutions that would be useful in the case of large amount of topics in a Pulsar cluster. Rate limiting on the /metrics endpoint doesn't sound like the correct approach. When

[DISCUSS] PIP-204: Reactive Java client for Apache Pulsar

2022-08-29 Thread Lari Hotari
Hi all, I have drafted PIP-204: Reactive Java client for Apache Pulsar. PIP link: https://github.com/apache/pulsar/issues/17335 Here's a copy of the contents of the GH issue for your references: Motivation There's a need to "go reactive from end-to-end" when building modern reactive applicatio

Re: [DISCUSS] PIP-204: Reactive Java client for Apache Pulsar

2022-08-29 Thread Lari Hotari
I updated it to be PIP-205 since there was a previous reference of PIP-204. :) -Lari On 2022/08/29 12:55:43 Lari Hotari wrote: > Hi all, > > I have drafted PIP-204: Reactive Java client for Apache Pulsar. > > PIP link: > https://github.com/apache/pulsar/issues/17335 > &

Re: Pulsar CI congested, master branch build broken

2022-08-29 Thread Lari Hotari
master branch is broken once again. Here's the fix: https://github.com/apache/pulsar/pull/17339 Please review and merge -Lari On 2022/08/26 12:00:20 Lari Hotari wrote: > Hi, > > GitHub Actions builds have been piling up in the build queue in the last few > days.

Re: [DISCUSS] Move PIPs to the codebase?

2022-08-29 Thread Lari Hotari
Is there a specific reason to put the PIPs in the apache/pulsar repository? I think that it will add unnecessary cruft to our core source code repository. Could we instead create a separate repository to hold the PIP files? The example of Rust lang has a separate repository too, https://github.

Re: Pulsar CI congested, master branch build broken

2022-08-30 Thread Lari Hotari
. Hopefully we get the CI slowness issue solved asap. BR, Lari On 2022/08/26 12:00:20 Lari Hotari wrote: > Hi, > > GitHub Actions builds have been piling up in the build queue in the last few > days. > I posted on bui...@apache.org > https://lists.apache.org/thread/6lbqr0f6

[VOTE] PIP-205: Reactive Java client for Apache Pulsar

2022-08-31 Thread Lari Hotari
Hello, This is the official VOTE thread for PIP-205: Reactive Java client for Apache Pulsar. PIP link: https://github.com/apache/pulsar/issues/17335 discussion: https://lists.apache.org/thread/xkfl5px7qgt5rrxw5pj0g318r6mbdlz1 Given the proposal is accepted, as the next step I will proceed to req

Re: [VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder

2022-08-31 Thread Lari Hotari
+1 (binding) -Lari On 2022/08/29 13:25:51 tison wrote: > Hi devs, > > This is the official thread VOTE for PIP-200 Package Pulsar Trino distro > and config in a dedicated folder. > > Here is the PIP issue: https://github.com/apache/pulsar/issues/17137 > Here is the discussion thread: > https://

[DISCUSS] Call to improve Pulsar contributor's experience

2022-09-01 Thread Lari Hotari
Hi, I think that we would need to improve the experience for contributors. It's currently a big challenge to get a PR to the state where tests pass, mainly because of the large amount of flaky tests and frequent congestions in Pulsar CI. We don't tell this to the contributors in the PR template

Re: [DISCUSS] Call to improve Pulsar contributor's experience

2022-09-01 Thread Lari Hotari
I'm sharing my experience about finding the reason why the build fails. This has been frustrating. In the past, it has been a challenge to find the reason why the build fails. Not that we don't have the information about the failure, but the challenge is that there's too much information. :) L

Re: [DISCUSS] Call to improve Pulsar contributor's experience

2022-09-01 Thread Lari Hotari
On 2022/09/01 08:36:11 Yu wrote: > # 1 > For pure doc PRs (only update .md files), do they run the same tests as > code PRs? > If so, can we set them to run only doc-related tests and skip code tests > (since they're easily failed)? > In this way, docs can be iterated faster. The solution is alrea

  1   2   3   4   5   6   7   8   >