Re: [VOTE] Pulsar Release 2.7.5 Candidate 1
+1 On 2022/02/11 13:47:50 Lari Hotari wrote: > This is the first release candidate for Apache Pulsar, version 2.7.5. > > It contains the following commits after the previous release: > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-1 > > *** Please download, test and vote on this release. This vote will stay open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-1/ > > SHA-512 checksums: > > b0d4e8c05870e0fe8cb50a80e15811b59972fc7f1d3fa93fa6cf2b61797176e80b434334aa67a5619de2a46a6ae9e51286dad524cac7af76239cda790d3574cc > apache-pulsar-2.7.5-bin.tar.gz > e9a88d6847828fdb051fe2a4663e5b66beb0a066e9e170682df4e0330fe5d59d27c88908f493e2100f35a2469a8f77a97c94da752aac320bf4413aaed57570d6 > apache-pulsar-2.7.5-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1138/ > > The tag to be voted upon: > v2.7.5-candidate-1 (db8761ebb370db1ae731a807afb583ac346378fe) > https://github.com/apache/pulsar/releases/tag/v2.7.5-candidate-1 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build > and run the Pulsar standalone service. >
[GitHub] [pulsar-client-node] tuteng commented on pull request #194: Fixed windows install client
tuteng commented on pull request #194: URL: https://github.com/apache/pulsar-client-node/pull/194#issuecomment-1041259278 > Should we update the document as well? I think we need to define `OS_ARCH` environment variable when we compile the node client. Yes I think the documentation needs to be updated, I also think os_arch is an optional field, if not defined it will use this `\vcpkg_installed\bin\` to find dependencies -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
Ping Il Dom 13 Feb 2022, 11:49 Enrico Olivelli ha scritto: > Hello, > I have a use case in which my application MUST use a non-partitioned > topic to work properly, but the topic name is (of course) configurable > by the user. > If I am not using a non-partitioned topic, I do not have string > guarantees on the ordering of the message (because messages will be > spread across multiple partitions). > > Currently there is no way to require that the topic IS a > NON-PARTITIONED topic. Especially when the topic does not exist yet, > you configured the namespace to create partitioned topics by default. > > Using a PulsarAdmin preliminary call is not a good workaround because > it is expensive, and ideally I want to be able to create the Producer > (or the Consumer) and see that everything works automatically, failing > in case of a partitioned topic. > > Thoughts ? > > > Enrico >
Re: [VOTE] Pulsar Release 2.7.5 Candidate 1
The branch-2.7 build didn't complete cleanly for the release commit https://github.com/apache/pulsar/commit/ed40840a65c413202c0aab6b9b1a29ed370f6fb7 . (click on the red cross icon before the text "Release 2.7.5" to see the build results popup) There are 3 failing build jobs: - 2 for "CI - Build - Multiple - OS / build" - "CI - Integration - Cli / cli" It's safe to ignore the failures for "CI - Build - Multiple - OS / build" since that doesn't run tests and the build was also failing in the 2.7.4 release commit: https://github.com/apache/pulsar/commit/ab451b855d873a9bad2005f939a23118a583baa9 (click on the red cross icon to see build results) I am running the failing "CI - Integration - Cli / cli" build job locally with the command "mvn -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-cli.xml -DintegrationTests -DredirectTestOutputToFile=false" (after building the 2.7.5 docker images and docker test image locally). The test doesn't complete cleanly. This is the error message in integration tests: [integration] [INFO] [stdout] org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available [integration] [INFO] [stdout] 08:20:16.613 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN org.apache.pulsar.broker.service.ServerCnx - [/172.19.0.8:54068][persistent://compaction-test-rest-wmxw/ns1/topic1][sub1] Failed to create consumer: consumerId=0, org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available I can see this error message in the bookies (docker exec -it CLITest-gmwag-pulsar-bookie-1 cat /var/log/pulsar/bookie.log) 08:10:20.616 [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, exiting bookie io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 536870912, max: 536870912) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:802) ~[io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final] This is the flaky test issue https://github.com/apache/pulsar/issues/9622 fixed by PR https://github.com/apache/pulsar/pull/9623 which was missing from branch-2.7 . After cherry-picking PR 9623, I can run CLI integration tests locally. There were 3 test failures: - org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest#testDeleteCluster java.lang.AssertionError: expected [0] but found [18] at org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest.testDeleteCluster(ClusterMetadataTearDownTest.java:122) - org.apache.pulsar.tests.integration.cli.CLITest#testCreateSubscriptionCommand org.apache.pulsar.tests.integration.docker.ContainerExecException: /pulsar/bin/pulsar-admin topics create-subscription persistent://public/default/testCreateSubscriptionCommmand --subscription subscription-0 failed on e03e5466c18da298a834ddbeeac9e6c80e583c27b28ecbac0bd472f148696a1b with error code 1 - org.apache.pulsar.tests.integration.compaction.TestCompaction#testPublishCompactAndConsumeCLI org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.tests.integration.compaction.TestCompaction.testPublishCompactAndConsumeCLI() didn't finish within the time-out 30 I'll run the integration tests locally for 2.7.4 release to see if there's a regression or whether it's just the tests that are flaky. -Lari On 2022/02/15 13:54:18 PengHui Li wrote: > Hi Lari > > The PR you shared https://github.com/apache/pulsar/pull/14240 does not run > any tests. > All the tests were completed in 1 min, please check. > > Thanks, > Penghui > > > > On Tue, Feb 15, 2022 at 7:23 PM Lari Hotari wrote: > > > The tests passed for the a PR based on revision a27e0853bda in branch-2.7 > > in this PR that I made to run the tests: > > https://github.com/apache/pulsar/pull/14240 . rev a27e0853bda is the last > > commit before the 2.7.5 release commit in branch-2.7 . > > > > I'll count your vote as +1 since all tests have passed. > > > > -Lari > > > > On 2022/02/15 00:36:28 PengHui Li wrote: > > > - Verified the signature > > > - Start the standalone and checked message publish and consumption > > > - Validate Cassandra connector > > > - Validate stateful function > > > > > > Hi Lari, I have checked the CI status for branch-2.7 > > > https://github.com/apache/pulsar/commits/branch-2.7 > > > There are 4 failed tests, could you please help check the failed test and > > > make sure > > > the tests get passed for 2.7.5? Since we don't run any checks before or > > > after cherry-picking, > > > I usually check the CI after cherry-picking, if there are failed tests, > > > need to confirm if it related to breaking changes > > > or a flaky test to retry. > > > > > > If all tests pass, please treat my validation as +1 > > > > > > Thanks, > > > Penghui > > > > > > On Mon, Feb 14, 2022 at 9:12 PM Haiting Jiang > > > wrote: > > > > > > > +1 > > > > > > > > Checks: > > > > > >
Re: [ANNOUNCE] New Committer: Li Li
Configurations! Thanks, Guangning Jia Zhai 于2022年2月16日周三 15:40写道： > Congratulations > > > On Wed, Feb 16, 2022 at 11:39 AM Yu wrote: > > > Hi Lili, > > > > Congrats! Well deserved! You've made a lot of great changes to the Pulsar > > website! > > > > On Tue, Feb 15, 2022 at 5:56 PM ZhangJian He wrote: > > > > > Congratulations > > > > > > Thanks > > > ZhangJian He > > > > > > Huanli Meng 于2022年2月15日周二 17:03写道： > > > > > > > Congratulations Li. > > > > > > > > BR// Huanli > > > > > > > > > On Feb 15, 2022, at 3:06 PM, PengHui Li > wrote: > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Li > > Li > > > > > https://github.com/urfreespace to become a committer and we are > > > pleased > > > > to > > > > > announce that he has accepted. > > > > > > > > > > Li Li has done a great contribution to Pulsar Website, > documentation. > > > > > > > > > > Welcome and Congratulations, Li Li! > > > > > > > > > > Please join us in congratulating and welcoming Li Li onboard! > > > > > > > > > > Best Regards, > > > > > Penghui Li on behalf of the Pulsar PMC > > > > > > > > > > > > > >
Re: [VOTE] Pulsar Release 2.7.5 Candidate 1
I'll vote again with -1. The "CI - Integration - Cli / cli" build job passes cleanly for 2.7.4 . This means that it's very likely that there's a regression in 2.7.5 . I'll investigate the issue. -Lari On 2022/02/16 10:21:23 Lari Hotari wrote: > The branch-2.7 build didn't complete cleanly for the release commit > https://github.com/apache/pulsar/commit/ed40840a65c413202c0aab6b9b1a29ed370f6fb7 > . > (click on the red cross icon before the text "Release 2.7.5" to see the build > results popup) > > There are 3 failing build jobs: > - 2 for "CI - Build - Multiple - OS / build" > - "CI - Integration - Cli / cli" > > It's safe to ignore the failures for "CI - Build - Multiple - OS / build" > since that doesn't run tests and the build was also failing in the 2.7.4 > release commit: > https://github.com/apache/pulsar/commit/ab451b855d873a9bad2005f939a23118a583baa9 > (click on the red cross icon to see build results) > > I am running the failing "CI - Integration - Cli / cli" build job locally > with the command "mvn -f tests/pom.xml test > -DintegrationTestSuiteFile=pulsar-cli.xml -DintegrationTests > -DredirectTestOutputToFile=false" (after building the 2.7.5 docker images and > docker test image locally). > > The test doesn't complete cleanly. This is the error message in integration > tests: > [integration] [INFO] [stdout] > org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty > bookies available > [integration] [INFO] [stdout] 08:20:16.613 > [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN > org.apache.pulsar.broker.service.ServerCnx - > [/172.19.0.8:54068][persistent://compaction-test-rest-wmxw/ns1/topic1][sub1] > Failed to create consumer: consumerId=0, > org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty > bookies available > > I can see this error message in the bookies (docker exec -it > CLITest-gmwag-pulsar-bookie-1 cat /var/log/pulsar/bookie.log) > 08:10:20.616 [bookie-io-1-9] ERROR org.apache.bookkeeper.proto.BookieServer - > Unable to allocate memory, exiting bookie > io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 > byte(s) of direct memory (used: 536870912, max: 536870912) > at > io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:802) > ~[io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final] > > This is the flaky test issue https://github.com/apache/pulsar/issues/9622 > fixed by PR https://github.com/apache/pulsar/pull/9623 which was missing from > branch-2.7 . > After cherry-picking PR 9623, I can run CLI integration tests locally. There > were 3 test failures: > - > org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest#testDeleteCluster > java.lang.AssertionError: expected [0] but found [18] > at > org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest.testDeleteCluster(ClusterMetadataTearDownTest.java:122) > - > org.apache.pulsar.tests.integration.cli.CLITest#testCreateSubscriptionCommand > org.apache.pulsar.tests.integration.docker.ContainerExecException: > /pulsar/bin/pulsar-admin topics create-subscription > persistent://public/default/testCreateSubscriptionCommmand --subscription > subscription-0 failed on > e03e5466c18da298a834ddbeeac9e6c80e583c27b28ecbac0bd472f148696a1b with error > code 1 > - > org.apache.pulsar.tests.integration.compaction.TestCompaction#testPublishCompactAndConsumeCLI > org.testng.internal.thread.ThreadTimeoutException: Method > org.apache.pulsar.tests.integration.compaction.TestCompaction.testPublishCompactAndConsumeCLI() > didn't finish within the time-out 30 > > I'll run the integration tests locally for 2.7.4 release to see if there's a > regression or whether it's just the tests that are flaky. > > > -Lari > > > > > > On 2022/02/15 13:54:18 PengHui Li wrote: > > Hi Lari > > > > The PR you shared https://github.com/apache/pulsar/pull/14240 does not run > > any tests. > > All the tests were completed in 1 min, please check. > > > > Thanks, > > Penghui > > > > > > > > On Tue, Feb 15, 2022 at 7:23 PM Lari Hotari wrote: > > > > > The tests passed for the a PR based on revision a27e0853bda in branch-2.7 > > > in this PR that I made to run the tests: > > > https://github.com/apache/pulsar/pull/14240 . rev a27e0853bda is the last > > > commit before the 2.7.5 release commit in branch-2.7 . > > > > > > I'll count your vote as +1 since all tests have passed. > > > > > > -Lari > > > > > > On 2022/02/15 00:36:28 PengHui Li wrote: > > > > - Verified the signature > > > > - Start the standalone and checked message publish and consumption > > > > - Validate Cassandra connector > > > > - Validate stateful function > > > > > > > > Hi Lari, I have checked the CI status for branch-2.7 > > > > https://github.com/apache/pulsar/commits/branch-2.7 > > > > There are 4 failed tests, could you please help check the failed test > > > > and > > > > make sure > > > > the test
Re: [DISCUSS] Apache Pulsar 2.10.0 release
> The contributors found there are many places that might also have the same > problem. I'm trying to understand "the problem" even though I haven't received replies from any of the contributors. For example, my question in https://github.com/apache/pulsar/issues/14013#issuecomment-1033528348 has been ignored. In the case of the "make async" changes, it seems to be about migrating from the blocking Servlet API to the Asynchronous Servlet API. For example this change: https://github.com/apache/pulsar/pull/14188/files#diff-0a1b84bf6bf3128bd4fc80875a959888ac68b989474afe536ac7cdd75d594400 There are a lot of these "make async" changes: https://github.com/apache/pulsar/pulls?q=is%3Apr+make+async . In Jetty documentation, there's an explanation about Asynchronous Servlets: https://wiki.eclipse.org/Jetty/Feature/Continuations#Why_Asynchronous_Servlets_.3F It also explains the benefits of asynchronous servlets: "The servlet API (<=2.5) supports only a synchronous call style, so that any waiting that a servlet needs to do must be with blocking. Unfortunately this means that the thread allocated to the request must be held during that wait along with all its resources: kernel thread, stack memory and often pooled buffers, character converters, EE authentication context, etc. It is wasteful of system resources to hold these resources while waiting. Significantly better scalability and quality of service can be achieved if waiting is done asynchronously." You can achieve better resource efficiency and better scalability with asynchronous servlets. Do we have such problems in Apache Pulsar with the Servlet API? -Lari On 2022/02/15 14:51:59 Lari Hotari wrote: > On 2022/02/15 14:13:59 PengHui Li wrote: > > The rationale for these changes, I think it starts from this PR > > https://github.com/apache/pulsar/pull/13666 > > This is the only one example, we have seen the same issue again and again. > > After #13666 get merged, > > The contributors found there are many places that might also have the same > > problem. > > Thanks for replying, Penghui. The problem is that there is no rationale nor > description in that PR, https://github.com/apache/pulsar/pull/13666 . The > only sentence there is "Avoid call sync method in async rest API for delete > subscription". > > > "we have seen the same issue again and again." > > What issue did you see? Please share more context. Thanks for the patience. > > BR, Lari >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
On 2022/02/16 00:58:20 PengHui Li wrote: > Which is a sync method. Ultimately this could lead to all the pulsar-web > thread > blocked. we'd better not introduce blocking calls if we use AsyncResponse. > > > What issue did you see? Please share more context. Thanks for the > patience. > > It happened very earlier > > Here is the issue https://github.com/apache/pulsar/issues/4756 > And here is also a related fix https://github.com/apache/pulsar/pull/10619 Penghui, Thank you for the patience, and thanks for sharing more context. I happened to send a reply before reading your message, so please bear with me. So finally, I understand that "the problem" is that all HTTP server threads are blocked and this makes the Pulsar Admin API unavailable. To support the blocking servlet API, Jetty uses a default thread pool that can grow to up to 200 threads (https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) . However this default of 200 maximum threads is not used in Pulsar. The problem is that Pulsar uses a low value that assumes asynchronous API usage: https://github.com/apache/pulsar/blob/5c3ddc26d6e07eb0473b11b5ecc8318c1efe414b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L201-L204 Pulsar should be using a high value (for example 200) as long as there are blocking calls in Admin APIs. The mitigation to the issue of all HTTP server threads getting blocked is setting "numHttpServerThreads=200" in broker.conf. Regarding the "make async" changes, It is an optimization to migrate from the blocking servlet api to the asynchronous servlet api. This work isn't urgent since we can simply mitigate the HTTP server threads getting blocked by setting "numHttpServerThreads=200" in broker.conf. "the problem" will be resolved immediately without risks of regressions that are involved in making the sync -> async changes. Penghui, would you mind adding a GitHub issue for the problem where all HTTP threads get blocked and the Pulsar Admin API stops responding? I can follow up with a PR which updates the default for numHttpServerThreads to 200. This is a maximum value and Jetty starts with 8 threads. We can agree on the default value to use in the PR. Thank you for the great collaboration on sharing the context and describing the problem patiently. BR, -Lari
Re: [DISCUSS] Apache Pulsar 2.10.0 release
I created PR https://github.com/apache/pulsar/pull/14320 to set numHttpServerThreads=200 . Please review On 2022/02/16 12:39:34 Lari Hotari wrote: > On 2022/02/16 00:58:20 PengHui Li wrote: > > Which is a sync method. Ultimately this could lead to all the pulsar-web > > thread > > blocked. we'd better not introduce blocking calls if we use AsyncResponse. > > > > > What issue did you see? Please share more context. Thanks for the > > patience. > > > > It happened very earlier > > > > Here is the issue https://github.com/apache/pulsar/issues/4756 > > And here is also a related fix https://github.com/apache/pulsar/pull/10619 > > Penghui, Thank you for the patience, and thanks for sharing more context. I > happened to send a reply before reading your message, so please bear with me. > > So finally, I understand that "the problem" is that all HTTP server threads > are blocked and this makes the Pulsar Admin API unavailable. > > To support the blocking servlet API, Jetty uses a default thread pool that > can grow to up to 200 threads > (https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > . > However this default of 200 maximum threads is not used in Pulsar. > > The problem is that Pulsar uses a low value that assumes asynchronous API > usage: > https://github.com/apache/pulsar/blob/5c3ddc26d6e07eb0473b11b5ecc8318c1efe414b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L201-L204 > Pulsar should be using a high value (for example 200) as long as there are > blocking calls in Admin APIs. > > The mitigation to the issue of all HTTP server threads getting blocked is > setting "numHttpServerThreads=200" in broker.conf. > > Regarding the "make async" changes, It is an optimization to migrate from the > blocking servlet api to the asynchronous servlet api. This work isn't urgent > since we can simply mitigate the HTTP server threads getting blocked by > setting "numHttpServerThreads=200" in broker.conf. "the problem" will be > resolved immediately without risks of regressions that are involved in making > the sync -> async changes. > > Penghui, would you mind adding a GitHub issue for the problem where all HTTP > threads get blocked and the Pulsar Admin API stops responding? > > I can follow up with a PR which updates the default for numHttpServerThreads > to 200. This is a maximum value and Jetty starts with 8 threads. We can agree > on the default value to use in the PR. > > Thank you for the great collaboration on sharing the context and describing > the problem patiently. > > BR, > > -Lari >
[GitHub] [pulsar-manager] sama8 commented on pull request #434: SAML Integration for Login
sama8 commented on pull request #434: URL: https://github.com/apache/pulsar-manager/pull/434#issuecomment-1041477673 Sharing if relevant -> [SAML Jackson](https://github.com/boxyhq/jackson) :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [ANNOUNCE] New Committer: Li Li
Congrats! Thanks, Haiting On 2022/02/16 10:51:52 Guangning E wrote: > Configurations! > > > Thanks, > Guangning > > Jia Zhai 于2022年2月16日周三 15:40写道： > > > Congratulations > > > > > > On Wed, Feb 16, 2022 at 11:39 AM Yu wrote: > > > > > Hi Lili, > > > > > > Congrats! Well deserved! You've made a lot of great changes to the Pulsar > > > website! > > > > > > On Tue, Feb 15, 2022 at 5:56 PM ZhangJian He wrote: > > > > > > > Congratulations > > > > > > > > Thanks > > > > ZhangJian He > > > > > > > > Huanli Meng 于2022年2月15日周二 17:03写道： > > > > > > > > > Congratulations Li. > > > > > > > > > > BR// Huanli > > > > > > > > > > > On Feb 15, 2022, at 3:06 PM, PengHui Li > > wrote: > > > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Li > > > Li > > > > > > https://github.com/urfreespace to become a committer and we are > > > > pleased > > > > > to > > > > > > announce that he has accepted. > > > > > > > > > > > > Li Li has done a great contribution to Pulsar Website, > > documentation. > > > > > > > > > > > > Welcome and Congratulations, Li Li! > > > > > > > > > > > > Please join us in congratulating and welcoming Li Li onboard! > > > > > > > > > > > > Best Regards, > > > > > > Penghui Li on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Hi lari, > So finally, I understand that "the problem" is that all HTTP server threads are blocked and this makes the Pulsar Admin API unavailable. To support the blocking servlet API, Jetty uses a default thread pool that can grow to up to 200 threads ( https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) . However this default of 200 maximum threads is not used in Pulsar. Regarding the "make async" changes, It is an optimization to migrate from the blocking servlet api to the asynchronous servlet api. This work isn't urgent since we can simply mitigate the HTTP server threads getting blocked by setting "numHttpServerThreads=200" in broker.conf. "the problem" will be resolved immediately without risks of regressions that are involved in making the sync -> async changes. Yes, this is the problem. But I am against using 200 threads as the max web server thread by default, it can't work for cases that the broker without that much memory, it will lead to more serious problems that the service quality of messaging API gets worse due to the JVM GC, even memory overflow. Yes, it isn't urgent. So I said it's not a blocker for the 2.10 release, and all the PRs are not cherry-picked to branch-2.x This is an optimization for pulsar, the current implementation does not use jetty async API well, we should fix it, we should reduce the code with bad smells, and using async API is also a more efficient way without opening such jetty threads. Do you have any concerns about the way the modification becomes purely async? > Penghui, would you mind adding a GitHub issue for the problem where all HTTP threads get blocked and the Pulsar Admin API stops responding? https://github.com/apache/pulsar/issues/4756 the attachment from the issue is a good example Regards, Penghui On Wed, Feb 16, 2022 at 9:04 PM Lari Hotari wrote: > I created PR https://github.com/apache/pulsar/pull/14320 to set > numHttpServerThreads=200 . > Please review > > On 2022/02/16 12:39:34 Lari Hotari wrote: > > On 2022/02/16 00:58:20 PengHui Li wrote: > > > Which is a sync method. Ultimately this could lead to all the > pulsar-web > > > thread > > > blocked. we'd better not introduce blocking calls if we use > AsyncResponse. > > > > > > > What issue did you see? Please share more context. Thanks for the > > > patience. > > > > > > It happened very earlier > > > > > > Here is the issue https://github.com/apache/pulsar/issues/4756 > > > And here is also a related fix > https://github.com/apache/pulsar/pull/10619 > > > > Penghui, Thank you for the patience, and thanks for sharing more > context. I happened to send a reply before reading your message, so please > bear with me. > > > > So finally, I understand that "the problem" is that all HTTP server > threads are blocked and this makes the Pulsar Admin API unavailable. > > > > To support the blocking servlet API, Jetty uses a default thread pool > that can grow to up to 200 threads ( > https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > . > > However this default of 200 maximum threads is not used in Pulsar. > > > > The problem is that Pulsar uses a low value that assumes asynchronous > API usage: > > > https://github.com/apache/pulsar/blob/5c3ddc26d6e07eb0473b11b5ecc8318c1efe414b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L201-L204 > > Pulsar should be using a high value (for example 200) as long as there > are blocking calls in Admin APIs. > > > > The mitigation to the issue of all HTTP server threads getting blocked > is setting "numHttpServerThreads=200" in broker.conf. > > > > Regarding the "make async" changes, It is an optimization to migrate > from the blocking servlet api to the asynchronous servlet api. This work > isn't urgent since we can simply mitigate the HTTP server threads getting > blocked by setting "numHttpServerThreads=200" in broker.conf. "the problem" > will be resolved immediately without risks of regressions that are involved > in making the sync -> async changes. > > > > Penghui, would you mind adding a GitHub issue for the problem where all > HTTP threads get blocked and the Pulsar Admin API stops responding? > > > > I can follow up with a PR which updates the default for > numHttpServerThreads to 200. This is a maximum value and Jetty starts with > 8 threads. We can agree on the default value to use in the PR. > > > > Thank you for the great collaboration on sharing the context and > describing the problem patiently. > > > > BR, > > > > -Lari > > >
Re: [VOTE] Pulsar Release 2.7.5 Candidate 1
Thanks for the great work! Penghui On Wed, Feb 16, 2022 at 6:54 PM Lari Hotari wrote: > I'll vote again with -1. > > The "CI - Integration - Cli / cli" build job passes cleanly for 2.7.4 . > This means that it's very likely that there's a regression in 2.7.5 . I'll > investigate the issue. > > -Lari > > On 2022/02/16 10:21:23 Lari Hotari wrote: > > The branch-2.7 build didn't complete cleanly for the release commit > > > https://github.com/apache/pulsar/commit/ed40840a65c413202c0aab6b9b1a29ed370f6fb7 > . > > (click on the red cross icon before the text "Release 2.7.5" to see the > build results popup) > > > > There are 3 failing build jobs: > > - 2 for "CI - Build - Multiple - OS / build" > > - "CI - Integration - Cli / cli" > > > > It's safe to ignore the failures for "CI - Build - Multiple - OS / > build" since that doesn't run tests and the build was also failing in the > 2.7.4 release commit: > > > https://github.com/apache/pulsar/commit/ab451b855d873a9bad2005f939a23118a583baa9 > > (click on the red cross icon to see build results) > > > > I am running the failing "CI - Integration - Cli / cli" build job > locally with the command "mvn -f tests/pom.xml test > -DintegrationTestSuiteFile=pulsar-cli.xml -DintegrationTests > -DredirectTestOutputToFile=false" (after building the 2.7.5 docker images > and docker test image locally). > > > > The test doesn't complete cleanly. This is the error message in > integration tests: > > [integration] [INFO] [stdout] > org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty > bookies available > > [integration] [INFO] [stdout] 08:20:16.613 > [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN > org.apache.pulsar.broker.service.ServerCnx - > [/172.19.0.8:54068][persistent://compaction-test-rest-wmxw/ns1/topic1][sub1] > Failed to create consumer: consumerId=0, > org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty > bookies available > > > > I can see this error message in the bookies (docker exec -it > CLITest-gmwag-pulsar-bookie-1 cat /var/log/pulsar/bookie.log) > > 08:10:20.616 [bookie-io-1-9] ERROR > org.apache.bookkeeper.proto.BookieServer - Unable to allocate memory, > exiting bookie > > io.netty.util.internal.OutOfDirectMemoryError: failed to allocate > 16777216 byte(s) of direct memory (used: 536870912, max: 536870912) > > at > io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:802) > ~[io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final] > > > > This is the flaky test issue > https://github.com/apache/pulsar/issues/9622 fixed by PR > https://github.com/apache/pulsar/pull/9623 which was missing from > branch-2.7 . > > After cherry-picking PR 9623, I can run CLI integration tests locally. > There were 3 test failures: > > - > org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest#testDeleteCluster > > java.lang.AssertionError: expected [0] but found [18] > > at > org.apache.pulsar.tests.integration.cli.ClusterMetadataTearDownTest.testDeleteCluster(ClusterMetadataTearDownTest.java:122) > > - > org.apache.pulsar.tests.integration.cli.CLITest#testCreateSubscriptionCommand > > org.apache.pulsar.tests.integration.docker.ContainerExecException: > /pulsar/bin/pulsar-admin topics create-subscription > persistent://public/default/testCreateSubscriptionCommmand --subscription > subscription-0 failed on > e03e5466c18da298a834ddbeeac9e6c80e583c27b28ecbac0bd472f148696a1b with error > code 1 > > - > org.apache.pulsar.tests.integration.compaction.TestCompaction#testPublishCompactAndConsumeCLI > > org.testng.internal.thread.ThreadTimeoutException: Method > org.apache.pulsar.tests.integration.compaction.TestCompaction.testPublishCompactAndConsumeCLI() > didn't finish within the time-out 30 > > > > I'll run the integration tests locally for 2.7.4 release to see if > there's a regression or whether it's just the tests that are flaky. > > > > > > -Lari > > > > > > > > > > > > On 2022/02/15 13:54:18 PengHui Li wrote: > > > Hi Lari > > > > > > The PR you shared https://github.com/apache/pulsar/pull/14240 does > not run > > > any tests. > > > All the tests were completed in 1 min, please check. > > > > > > Thanks, > > > Penghui > > > > > > > > > > > > On Tue, Feb 15, 2022 at 7:23 PM Lari Hotari > wrote: > > > > > > > The tests passed for the a PR based on revision a27e0853bda in > branch-2.7 > > > > in this PR that I made to run the tests: > > > > https://github.com/apache/pulsar/pull/14240 . rev a27e0853bda is > the last > > > > commit before the 2.7.5 release commit in branch-2.7 . > > > > > > > > I'll count your vote as +1 since all tests have passed. > > > > > > > > -Lari > > > > > > > > On 2022/02/15 00:36:28 PengHui Li wrote: > > > > > - Verified the signature > > > > > - Start the standalone and checked message publish and consumption > > > > > - Validate Cassandra connector > > > > > - Validate stateful function > > > > > > > > > > Hi Lari, I have checked the CI status
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Hi all Just put an update here. We have 2 PRs[1] https://github.com/apache/pulsar/pull/13376 and https://github.com/apache/pulsar/pull/13341 need to do the final verification, and you are also very welcome to verify these 2 changes in your environment, cases. I will build the release and start the vote before next Monday(GMT+8) Regards Penghui On Wed, Feb 16, 2022 at 10:22 PM PengHui Li wrote: > Hi lari, > > > So finally, I understand that "the problem" is that all HTTP server > threads are blocked and this makes the Pulsar Admin API unavailable. > > To support the blocking servlet API, Jetty uses a default thread pool that > can grow to up to 200 threads ( > https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > . > However this default of 200 maximum threads is not used in Pulsar. > > Regarding the "make async" changes, It is an optimization to migrate from > the blocking servlet api to the asynchronous servlet api. This work isn't > urgent since we can simply mitigate the HTTP server threads getting blocked > by setting "numHttpServerThreads=200" in broker.conf. "the problem" will be > resolved immediately without risks of regressions that are involved in > making the sync -> async changes. > > Yes, this is the problem. But I am against using 200 threads as the max > web server thread by default, > it can't work for cases that the broker without that much memory, it will > lead to more serious problems > that the service quality of messaging API gets worse due to the JVM > GC, even memory overflow. > > Yes, it isn't urgent. So I said it's not a blocker for the 2.10 release, > and all the PRs are not cherry-picked to branch-2.x > This is an optimization for pulsar, the current implementation does not > use jetty async API well, we should fix it, > we should reduce the code with bad smells, and using async API is also > a more efficient way without opening such jetty threads. > Do you have any concerns about the way the modification becomes purely > async? > > > Penghui, would you mind adding a GitHub issue for the problem where all > HTTP threads get blocked and the Pulsar Admin API stops responding? > > https://github.com/apache/pulsar/issues/4756 the attachment from the > issue is a good example > > Regards, > Penghui > > > On Wed, Feb 16, 2022 at 9:04 PM Lari Hotari wrote: > >> I created PR https://github.com/apache/pulsar/pull/14320 to set >> numHttpServerThreads=200 . >> Please review >> >> On 2022/02/16 12:39:34 Lari Hotari wrote: >> > On 2022/02/16 00:58:20 PengHui Li wrote: >> > > Which is a sync method. Ultimately this could lead to all the >> pulsar-web >> > > thread >> > > blocked. we'd better not introduce blocking calls if we use >> AsyncResponse. >> > > >> > > > What issue did you see? Please share more context. Thanks for the >> > > patience. >> > > >> > > It happened very earlier >> > > >> > > Here is the issue https://github.com/apache/pulsar/issues/4756 >> > > And here is also a related fix >> https://github.com/apache/pulsar/pull/10619 >> > >> > Penghui, Thank you for the patience, and thanks for sharing more >> context. I happened to send a reply before reading your message, so please >> bear with me. >> > >> > So finally, I understand that "the problem" is that all HTTP server >> threads are blocked and this makes the Pulsar Admin API unavailable. >> > >> > To support the blocking servlet API, Jetty uses a default thread pool >> that can grow to up to 200 threads ( >> https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) >> . >> > However this default of 200 maximum threads is not used in Pulsar. >> > >> > The problem is that Pulsar uses a low value that assumes asynchronous >> API usage: >> > >> https://github.com/apache/pulsar/blob/5c3ddc26d6e07eb0473b11b5ecc8318c1efe414b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L201-L204 >> > Pulsar should be using a high value (for example 200) as long as there >> are blocking calls in Admin APIs. >> > >> > The mitigation to the issue of all HTTP server threads getting blocked >> is setting "numHttpServerThreads=200" in broker.conf. >> > >> > Regarding the "make async" changes, It is an optimization to migrate >> from the blocking servlet api to the asynchronous servlet api. This work >> isn't urgent since we can simply mitigate the HTTP server threads getting >> blocked by setting "numHttpServerThreads=200" in broker.conf. "the problem" >> will be resolved immediately without risks of regressions that are involved >> in making the sync -> async changes. >> > >> > Penghui, would you mind adding a GitHub issue for the problem where all >> HTTP threads get blocked and the Pulsar Admin API stops responding? >> > >> > I can follow up with a PR which updates the defau
[GitHub] [pulsar-site] dave2wave opened a new pull request #1: [PIP-87][Website] Publish website from asf-site-next branch
dave2wave opened a new pull request #1: URL: https://github.com/apache/pulsar-site/pull/1 Motivation We are ready to switch over website publishing to the pulsar-site repository. Modifications Update the .asf.yaml file which to publish the website from the asf-site-next branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] Pulsar Client Go Release 0.8.0 Candidate 1
+1 Checked signatures, compiled and run perf tool. -- Matteo Merli On Tue, Feb 15, 2022 at 7:28 PM r...@apache.org wrote: > > Hi everyone, > Please review and vote on the release candidate #1 for the version > 0.8.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This is the first release candidate for Apache Pulsar Go client, version > 0.8.0. > > It fixes the following issues: > https://github.com/apache/pulsar-client-go/milestone/9?closed=1 > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download these packages and review this release candidate: > - Review release notes > - Download the source package (verify shasum, and asc) and follow the > README.md to build and run the pulsar-client-go. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Source file: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.8.0-candidate-1/ > > The tag to be voted upon: > v0.8.0 > https://github.com/apache/pulsar-client-go/tree/v0.8.0-candidate-1 > > SHA-512 checksums: > 0015cc735af13139a564f0fd57c58af753579953e3c12f08cb62d8e9801cdc2964c60392ed9a98dd54845be7f7b1a353bd49cecb03360b4cb691904f8447769c > apache-pulsar-client-go-0.8.0-src.tar.gz
[DISCUSS] PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
URL: https://github.com/apache/pulsar/issues/14329 Motivation Since Pulsar Admin API uses the blocking servlet API, all Jetty threads might be occupied and this causes unavailability of the Pulsar Admin API. The default value for the maximum number of threads for Jetty is too low in Pulsar. That is the root cause of many problems where Pulsar Admin API is unavailable when all threads are in use. Additional context - Examples of previous issues where Jetty threads have been occupied and caused problems: #13666 #4756 #10619 - Mailing list thread about “make async” changes: https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l Implementation - Jetty defaults to 200 maximum threads, to prevent thread pool starvation. Make Pulsar use the same default value by setting numHttpServerThreads=200. - Update the documentation for numHttpServerThreads - The PR is already in place: https://github.com/apache/pulsar/pull/14320 - Set Jetty selectors and acceptors parameters to -1 so that Jetty automatically chooses optimal values based on available cores. The rationale is explained in the Q&A below. - A separate PR will be made for this change. Q&A Q: What’s the reason of setting the default value to 200? If the node just have one core, what will happen? These are threads. Jetty defaults to 200 maximum threads, to prevent thread pool starvation. This is recommended when using blocking Servlet API. The problem is that Pulsar uses the blocking servlet API and doesn’t have a sufficient number of threads which are needed and recommended. The value 200 doesn’t mean that there will be 200 threads to start with. This is the maximum size for the thread pool. When the value is more than 8, Jetty will start with 8 initial threads and add more threads to the pool when all threads are occupied. Q: Do we need to take the number of system cores into consideration for the maximum threads of the thread pool? No. Jetty is different from Netty in this aspect. In Netty, everything should be asynchronous and “thou shall never block”. In Jetty, the maximum number of threads for the thread pool should be set to 50-500 threads and blocking operations are fine. The recommendation for the thread pool is explained in Jetty documentation https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool > Thread Pool > Configure with goal of limiting memory usage maximum available. Typically this is >50 and <500 However, there are separate settings which should take the number of available processors (cores) into account in Jetty. http port acceptor and selector count: https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L88 https port acceptor and selector count: https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L125 Jetty documentantion for acceptors: > Acceptors > The standard rule of thumb for the number of Accepters to configure is one per CPU on a given machine. Jetty documentation for selectors: > Selectors > The default number of selectors is equal to half of the number of processors available to the JVM, which should allow optimal performance even if all the connections used are performing significant non-blocking work in the callback tasks. The settings in jetty are the “acceptor” and “selector” thread count settings. These have been fixed to 1 in Pulsar. The acceptors and selectors settings should be both set to -1. Jetty would pick the recommended count based on cores in that case.
[VOTE] Pulsar Release 2.9.2 Candidate 3
This is the third release candidate for Apache Pulsar, version 2.9.2. *** Please download, test, and vote on this release. This vote will stay open for at least 72 hours *** Note that we are voting upon the source (tag), binaries are provided for convenience. Source and binary files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-3/ SHA-512 checksums: 21ab21df057dc181d02806e67a8d85b861fa60a8413a8e4058ffcb738d9ef81d8797c4e7249d50cf2a23c7437cb2880f4240d010a983eda33a2071f5fddf6b01 apache-pulsar-2.9.2-bin.tar.gz f927d77aba32d1112cd009a1d91a8fca27a49fcb20ea215a5792dc16e2b04bd822238c3f29dd81a7853e20062a0454e2abe608626b24e4e25ff06915b7a04785 apache-pulsar-2.9.2-src.tar.gz Maven staging repo: https://repository.apache.org/content/repositories/orgapachepulsar-1141/ The tag to be voted upon: v2.9.2-candidate-3 (5615809ffd68599ae7fec1be0679ee83171f42dc) https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-3 Pulsar's KEYS file containing PGP keys we use to sign the release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS Please download the source package, and follow the README to build and run the Pulsar standalone service.
Re: [DISCUSS] Apache Pulsar 2.10.0 release
> I will build the release and start the vote before next Monday(GMT+8) Penghui, is your current plan to create branch-2.10, create the release artifacts, and start a vote on them all within a few days of each other? > I'm doing my best to follow PIP 47, but when seeing a potential break > change, I need to confirm it. > After all the potential break changes have been confirmed and fixed, I will > start the vote thread. I think we should review our current release plan before we move forward as proposed above. PIP 47 explicitly says that a month before the release date, the release manager will cut branches [0]. We don't yet have a `branch-2.10`. PIP 47 also defines a period of time for a feature freeze and then a code freeze. We have not yet had either. I propose we create branch-2.10 now and simultaneously announce that we are past the feature freeze period. Then, we can start the 2 week period for bug fixes that precedes the code freeze, as PIP 47 prescribes. Then, in two weeks, we can produce the first release candidate (also in PIP 47). While some may feel "behind" in getting out the 2.10 release, our priority must be to give the community time to verify the stability of the release. Thanks, Michael [0] https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan On Wed, Feb 16, 2022 at 9:09 AM PengHui Li wrote: > > Hi all > > Just put an update here. > > We have 2 PRs[1] https://github.com/apache/pulsar/pull/13376 and > https://github.com/apache/pulsar/pull/13341 > need to do the final verification, and you are also very welcome to verify > these 2 changes in your environment, cases. > > I will build the release and start the vote before next Monday(GMT+8) > > Regards > Penghui > > On Wed, Feb 16, 2022 at 10:22 PM PengHui Li wrote: > > > Hi lari, > > > > > So finally, I understand that "the problem" is that all HTTP server > > threads are blocked and this makes the Pulsar Admin API unavailable. > > > > To support the blocking servlet API, Jetty uses a default thread pool that > > can grow to up to 200 threads ( > > https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > > . > > However this default of 200 maximum threads is not used in Pulsar. > > > > Regarding the "make async" changes, It is an optimization to migrate from > > the blocking servlet api to the asynchronous servlet api. This work isn't > > urgent since we can simply mitigate the HTTP server threads getting blocked > > by setting "numHttpServerThreads=200" in broker.conf. "the problem" will be > > resolved immediately without risks of regressions that are involved in > > making the sync -> async changes. > > > > Yes, this is the problem. But I am against using 200 threads as the max > > web server thread by default, > > it can't work for cases that the broker without that much memory, it will > > lead to more serious problems > > that the service quality of messaging API gets worse due to the JVM > > GC, even memory overflow. > > > > Yes, it isn't urgent. So I said it's not a blocker for the 2.10 release, > > and all the PRs are not cherry-picked to branch-2.x > > This is an optimization for pulsar, the current implementation does not > > use jetty async API well, we should fix it, > > we should reduce the code with bad smells, and using async API is also > > a more efficient way without opening such jetty threads. > > Do you have any concerns about the way the modification becomes purely > > async? > > > > > Penghui, would you mind adding a GitHub issue for the problem where all > > HTTP threads get blocked and the Pulsar Admin API stops responding? > > > > https://github.com/apache/pulsar/issues/4756 the attachment from the > > issue is a good example > > > > Regards, > > Penghui > > > > > > On Wed, Feb 16, 2022 at 9:04 PM Lari Hotari wrote: > > > >> I created PR https://github.com/apache/pulsar/pull/14320 to set > >> numHttpServerThreads=200 . > >> Please review > >> > >> On 2022/02/16 12:39:34 Lari Hotari wrote: > >> > On 2022/02/16 00:58:20 PengHui Li wrote: > >> > > Which is a sync method. Ultimately this could lead to all the > >> pulsar-web > >> > > thread > >> > > blocked. we'd better not introduce blocking calls if we use > >> AsyncResponse. > >> > > > >> > > > What issue did you see? Please share more context. Thanks for the > >> > > patience. > >> > > > >> > > It happened very earlier > >> > > > >> > > Here is the issue https://github.com/apache/pulsar/issues/4756 > >> > > And here is also a related fix > >> https://github.com/apache/pulsar/pull/10619 > >> > > >> > Penghui, Thank you for the patience, and thanks for sharing more > >> context. I happened to send a reply before reading your message, so please > >> bear with me. > >> > > >> > So finally, I understand that "the problem" is that all HTTP server > >> threads are blocked and this makes the Pulsar Admin API unavailable. > >>
Re: [VOTE] Pulsar Release 2.8.3 Candidate 1
I agree that it can be non-trivial to determine if an issue is an internal test or an upstream issue. My main point is that we should prefer early communication of potential issues that affect pending releases and accept that the issue might prove to be nothing more than a short delay in the release cycle. The alternative is that a release manager completes the build process, which takes several hours, then discovers a new candidate is needed. There is still an inherent race condition for release creation and validation, but early communication (and collaboration) might help us avoid some of the cases where candidates are rejected immediately. For what it's worth, I don't think you need to verify your test correctness 100% before sharing potential issues that affect pending releases. A quick note to the dev mailing list indicating that you're investigating a potential issue would be sufficient and valuable. Thanks, Michael On Tue, Feb 15, 2022 at 6:30 PM PengHui Li wrote: > > > Just a friendly reminder that we are an open source project and other > community members who may have different internal testing may find better > fixes. > > Sorry, dave. I know we are an open source project. > We found a problem in our internal test, we need to confirm it is not > the test issue right? > > Do you suggest that we discuss this in the email when we find our internal > test errors? > The community has no responsibility to deal with our internal testing bugs. > > On Wed, Feb 16, 2022 at 8:04 AM Dave Fisher wrote: > > > > > > > > On Feb 15, 2022, at 3:59 PM, PengHui Li wrote: > > > > > >> I wonder if we should be more willing to share suspicions on the > > > mailing list when releases are actively being prepared. I would > > > have appreciated the notice, and I could have possibly helped > > > confirm the issue. On the other hand, I respect that it is hard to > > > know when to share a theory and when to test it out a bit more. > > > > > > Yes, It's just that these issues are happening in our internal testing > > > and we need to 100% confirm the issue and then share it out, > > > otherwise, We may confuse the contributors due to internal test problems. > > > > Just a friendly reminder that we are an open source project and other > > community members who may have different internal testing may find better > > fixes. > > > > > > > > This kind of problem is often easy to fix, but difficult to troubleshoot. > > > So usually when you are 100% confirmed the issue, usually know how to fix > > > it. > > > > > >> I think these were not included because they were labeled after I > > > cherry picked PRs on Friday. I will take a look and try to cherry pick > > > all of those before tagging the next release candidate > > > > > > Oh, I see. Thanks. I have checked > > > https://github.com/apache/pulsar/commits/branch-2.8 > > > LGTM. > > > > > > Regards, > > > Penghui > > > > > > On Wed, Feb 16, 2022 at 1:21 AM Michael Marshall > > > wrote: > > > > > >> Closing the vote with one -1 from Penghui. > > >> > > >>> We just confirmed yesterday that this is a breaking change, > > >>> I was just suspicious before, so do not share the information > > >>> to avoid confusing reviewers. For now, we confirmed it > > >> > > >> I wonder if we should be more willing to share suspicions on the > > >> mailing list when releases are actively being prepared. I would > > >> have appreciated the notice, and I could have possibly helped > > >> confirm the issue. On the other hand, I respect that it is hard to > > >> know when to share a theory and when to test it out a bit more. > > >> > > >>> And not all the PRs get cherry-picked to 2.8.3? > > >> > > >> I think these were not included because they were labeled after I > > >> cherry picked PRs on Friday. I will take a look and try to cherry pick > > >> all of those before tagging the next release candidate. > > >> > > >> Thanks, > > >> Michael > > >> > > >> > > >> > > >> > > >> On Tue, Feb 15, 2022 at 12:52 AM PengHui Li wrote: > > >>> > > >>> Sorry Michael > > >>> > > >>> There is a breaking change introduced in branch-2.8. > > >>> Sorry for that I forget to update the 2.8.3 release process, > > >>> Only update the context in 2.9.2 and 2.10.0. > > >>> > > >>> We just confirmed yesterday that this is a breaking change, > > >>> I was just suspicious before, so do not share the information > > >>> to avoid confusing reviewers. For now, we confirmed it > > >>> > > >>> The fixed PR is https://github.com/apache/pulsar/pull/14288 > > >>> > > >>> And not all the PRs get cherry-picked to 2.8.3? > > >>> Please check the list > > >>> > > >> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.8.3+is%3Aclosed+-label%3Acherry-picked%2Fbranch-2.8+ > > >>> If they are not a requirement for 2.8.3, we can move it to the next > > >> version. > > >>> For https://github.com/apache/pulsar/pull/14246, I can confirm, > > without > > >>> this fix, we will also introduce breaking change in 2.8.3 >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Michael, as we chatted in last weekly meeting (though not yet formalized), since we have never really done a feature freeze on the branch during paste releases, we should start from the next release, to give a decent preview of what to expect to developers in terms of dates. > While some may feel "behind" in getting out the 2.10 release, our > priority must be to give the community time to verify the stability of > the release. I don't think that changes the picture here. There are *always* last minute issues being discovered, and there is a call to be made on a case by case. The feature freeze will reduce the likelihood of introducing *more* issues by getting it from the master branch, but won't change a comma from issues that were already there. -- Matteo Merli On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall wrote: > > > I will build the release and start the vote before next Monday(GMT+8) > > Penghui, is your current plan to create branch-2.10, create the > release artifacts, and start a vote on them all within a few days of > each other? > > > I'm doing my best to follow PIP 47, but when seeing a potential break > > change, I need to confirm it. > > After all the potential break changes have been confirmed and fixed, I will > > start the vote thread. > > I think we should review our current release plan before we move > forward as proposed above. PIP 47 explicitly says that a month before > the release date, the release manager will cut branches [0]. We don't > yet have a `branch-2.10`. PIP 47 also defines a period of time for a > feature freeze and then a code freeze. We have not yet had either. > > I propose we create branch-2.10 now and simultaneously announce that > we are past the feature freeze period. Then, we can start the 2 week > period for bug fixes that precedes the code freeze, as PIP 47 > prescribes. Then, in two weeks, we can produce the first release > candidate (also in PIP 47). > > While some may feel "behind" in getting out the 2.10 release, our > priority must be to give the community time to verify the stability of > the release. > > Thanks, > Michael > > [0] https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan > > > > > On Wed, Feb 16, 2022 at 9:09 AM PengHui Li wrote: > > > > Hi all > > > > Just put an update here. > > > > We have 2 PRs[1] https://github.com/apache/pulsar/pull/13376 and > > https://github.com/apache/pulsar/pull/13341 > > need to do the final verification, and you are also very welcome to verify > > these 2 changes in your environment, cases. > > > > I will build the release and start the vote before next Monday(GMT+8) > > > > Regards > > Penghui > > > > On Wed, Feb 16, 2022 at 10:22 PM PengHui Li wrote: > > > > > Hi lari, > > > > > > > So finally, I understand that "the problem" is that all HTTP server > > > threads are blocked and this makes the Pulsar Admin API unavailable. > > > > > > To support the blocking servlet API, Jetty uses a default thread pool that > > > can grow to up to 200 threads ( > > > https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ExecutorThreadPool.java#L57) > > > . > > > However this default of 200 maximum threads is not used in Pulsar. > > > > > > Regarding the "make async" changes, It is an optimization to migrate from > > > the blocking servlet api to the asynchronous servlet api. This work isn't > > > urgent since we can simply mitigate the HTTP server threads getting > > > blocked > > > by setting "numHttpServerThreads=200" in broker.conf. "the problem" will > > > be > > > resolved immediately without risks of regressions that are involved in > > > making the sync -> async changes. > > > > > > Yes, this is the problem. But I am against using 200 threads as the max > > > web server thread by default, > > > it can't work for cases that the broker without that much memory, it will > > > lead to more serious problems > > > that the service quality of messaging API gets worse due to the JVM > > > GC, even memory overflow. > > > > > > Yes, it isn't urgent. So I said it's not a blocker for the 2.10 release, > > > and all the PRs are not cherry-picked to branch-2.x > > > This is an optimization for pulsar, the current implementation does not > > > use jetty async API well, we should fix it, > > > we should reduce the code with bad smells, and using async API is also > > > a more efficient way without opening such jetty threads. > > > Do you have any concerns about the way the modification becomes purely > > > async? > > > > > > > Penghui, would you mind adding a GitHub issue for the problem where all > > > HTTP threads get blocked and the Pulsar Admin API stops responding? > > > > > > https://github.com/apache/pulsar/issues/4756 the attachment from the > > > issue is a good example > > > > > > Regards, > > > Penghui > > > > > > > > > On Wed, Feb 16, 2022 at 9:04 PM Lari Hotari wrote: > > > > > >> I created PR https:
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
I don't think we should expose this feature flag on the client-side. Ideally, people shouldn't be aware of partitioned vs non-partitioned. - Sijie On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli wrote: > Ping > > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli ha > scritto: > > > Hello, > > I have a use case in which my application MUST use a non-partitioned > > topic to work properly, but the topic name is (of course) configurable > > by the user. > > If I am not using a non-partitioned topic, I do not have string > > guarantees on the ordering of the message (because messages will be > > spread across multiple partitions). > > > > Currently there is no way to require that the topic IS a > > NON-PARTITIONED topic. Especially when the topic does not exist yet, > > you configured the namespace to create partitioned topics by default. > > > > Using a PulsarAdmin preliminary call is not a good workaround because > > it is expensive, and ideally I want to be able to create the Producer > > (or the Consumer) and see that everything works automatically, failing > > in case of a partitioned topic. > > > > Thoughts ? > > > > > > Enrico > > >
[GitHub] [pulsar-helm-chart] bsheltonihs opened a new issue #229: Update pulsar-manager-service template to allow from admins to open multiple ports
bsheltonihs opened a new issue #229:
URL: https://github.com/apache/pulsar-helm-chart/issues/229
**Is your feature request related to a problem? Please describe.**
I am using argo-cd to deploy on openshift using these helm charts. The issue
I have is the backend admin service port is `7750` which the
`pulsar-manager-service.yaml` template doesn't expose. I am trying to follow
the instructions here:
[set-administrator-account-and-password](https://pulsar.apache.org/docs/en/administration-pulsar-manager/#set-administrator-account-and-password)
from a node in the cluster but it doesn't work. When I go into openshift and
manually add the `7750` port to the service I can then set the superuser info.
**Describe the solution you'd like**
I would like the `pulsar-manager-service.yaml` file to be updated from:
`ports:
- name: server
port: {{ .Values.pulsar_manager.service.port }}
targetPort: {{ .Values.pulsar_manager.service.targetPort }}
protocol: TCP`
to:
`ports:
- name: server
port: {{ .Values.pulsar_manager.service.port }}
targetPort: {{ .Values.pulsar_manager.service.targetPort }}
protocol: TCP
{{- if .Values.pulsar_manager.service.admin_port }}
- name: {{ .Values.pulsar_manager.service.admin_port.name }}
port: {{ .Values.pulsar_manager.service.admin_port.port }}
targetPort: {{ .Values.pulsar_manager.service.admin_port.targetPort }}
protocol: TCP
{{- end }}`
**Describe alternatives you've considered**
The only other options I have are:
1. Create and maintain my own helm charts (which long term isn't viable)
2. Do this manually (which doesn't allow it to be automated like I need)
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-helm-chart] bsheltonihs opened a new issue #230: issues creating clusterrole for functions
bsheltonihs opened a new issue #230:
URL: https://github.com/apache/pulsar-helm-chart/issues/230
**Is your feature request related to a problem? Please describe.**
I am using argo-cd to deploy on openshift using these helm charts. The issue
I have is when I enable functions the `broker-rbac` tries to create
`ClusterRole` and `ClusterRoleBinding`. The error I get is `Cluster level
ClusterRoleBinding "pulsar-functions-worker" can not be managed when in
namespaced mode` . This is because at openshift is managed at the enterprise
level and they don't allow this. However, you guys have addressed this same
issue with the `broker-role` and `broker-rolebinding` but adding the `{{- if
.Values.rbac.limit_to_namespace }}` logic in the
`broker-cluster-role-binding.yaml` template.
**Describe the solution you'd like**
I would like the same logic `{{- if .Values.rbac.limit_to_namespace }}` used
for the `broker-role` in the template `broker-cluster-role-binding.yaml` file
to be applied to the `function` section of the template `broker-rbac.yaml` file
**Describe alternatives you've considered**
The only other clear alternative that I see is set `resource.exclusions`
within argo-cd to exclude `ClusterRole` and `ClusterRoleBinding`. Then create
my own chart to create the `role` and `rolebinding` (Which isn't viable long
term)
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [DISCUSS] Apache Pulsar 2.10.0 release
When we discussed the code freeze in the community meeting on 2/3, I was under the impression that it was a new development to our existing release process. I subsequently learned it was already defined in PIP 47. Even if we haven't been following this part of PIP 47, what is the value in waiting until 2.11 to follow our already defined process? While I agree it is helpful to provide guidance on when a version will ship, I think it is more important to give the community time to test a release, even if that means we're a little late on our release schedule. So far, we haven't even created a branch to begin testing. Note also that Sijie suggested using a feature freeze early on in this thread. The 2.9.0 release is relevant here. It had 4 release candidates over 4 weeks and the final result was broken. That indicates to me that tagging an RC early does not guarantee an early release and that our current process isn't optimal and likely needs adjustments. I do not think we should wait to address these issues. I propose we start following PIP 47's guidance on code freeze and release stabilization periods. > I don't think that changes the picture here. There are *always* last > minute issues being discovered, and there is a call to be made on a > case by case. The feature freeze will reduce the likelihood of > introducing *more* issues by getting it from the master branch, but > won't change a comma from issues that were already there. I thought you wanted to implement a code/feature freeze to allow for more release stabilization. Can you clarify what you mean here? Thanks, Michael On Wed, Feb 16, 2022 at 2:42 PM Matteo Merli wrote: > > Michael, as we chatted in last weekly meeting (though not yet > formalized), since we have never really done a feature freeze on the > branch during paste releases, we should start from the next release, > to give a decent preview of what to expect to developers in terms of > dates. > > > While some may feel "behind" in getting out the 2.10 release, our > > priority must be to give the community time to verify the stability of > > the release. > > I don't think that changes the picture here. There are *always* last > minute issues being discovered, and there is a call to be made on a > case by case. The feature freeze will reduce the likelihood of > introducing *more* issues by getting it from the master branch, but > won't change a comma from issues that were already there. > > > > > -- > Matteo Merli > > > On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall > wrote: > > > > > I will build the release and start the vote before next Monday(GMT+8) > > > > Penghui, is your current plan to create branch-2.10, create the > > release artifacts, and start a vote on them all within a few days of > > each other? > > > > > I'm doing my best to follow PIP 47, but when seeing a potential break > > > change, I need to confirm it. > > > After all the potential break changes have been confirmed and fixed, I > > > will > > > start the vote thread. > > > > I think we should review our current release plan before we move > > forward as proposed above. PIP 47 explicitly says that a month before > > the release date, the release manager will cut branches [0]. We don't > > yet have a `branch-2.10`. PIP 47 also defines a period of time for a > > feature freeze and then a code freeze. We have not yet had either. > > > > I propose we create branch-2.10 now and simultaneously announce that > > we are past the feature freeze period. Then, we can start the 2 week > > period for bug fixes that precedes the code freeze, as PIP 47 > > prescribes. Then, in two weeks, we can produce the first release > > candidate (also in PIP 47). > > > > While some may feel "behind" in getting out the 2.10 release, our > > priority must be to give the community time to verify the stability of > > the release. > > > > Thanks, > > Michael > > > > [0] https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan > > > > > > > > > > On Wed, Feb 16, 2022 at 9:09 AM PengHui Li wrote: > > > > > > Hi all > > > > > > Just put an update here. > > > > > > We have 2 PRs[1] https://github.com/apache/pulsar/pull/13376 and > > > https://github.com/apache/pulsar/pull/13341 > > > need to do the final verification, and you are also very welcome to verify > > > these 2 changes in your environment, cases. > > > > > > I will build the release and start the vote before next Monday(GMT+8) > > > > > > Regards > > > Penghui > > > > > > On Wed, Feb 16, 2022 at 10:22 PM PengHui Li wrote: > > > > > > > Hi lari, > > > > > > > > > So finally, I understand that "the problem" is that all HTTP server > > > > threads are blocked and this makes the Pulsar Admin API unavailable. > > > > > > > > To support the blocking servlet API, Jetty uses a default thread pool > > > > that > > > > can grow to up to 200 threads ( > > > > https://github.com/eclipse/jetty.project/blob/4a0c91c0be53805e3fcffdcdcc9587d5301863db/jetty-util/src/main/jav
Re: [VOTE] Pulsar Release 2.8.3 Candidate 1
> I agree that it can be non-trivial to determine if an issue is an internal test or an upstream issue. My main point is that we should prefer early communication of potential issues that affect pending releases and accept that the issue might prove to be nothing more than a short delay in the release cycle. The alternative is that a release manager completes the build process, which takes several hours, then discovers a new candidate is needed. There is still an inherent race condition for release creation and validation, but early communication (and collaboration) might help us avoid some of the cases where candidates are rejected immediately. I have explained here https://lists.apache.org/thread/m5so1scfok44hnd74436w100smn2vbh3 at 11/02/2022 Have you seen it? As I said before, I forget to update this thread, I have declared at the first time that I respond to this thread. it's not that I don't want to share to this thread. I can't accept dave and you thinking I don't want to share here. it's completely different > For what it's worth, I don't think you need to verify your test correctness 100% before sharing potential issues that affect pending releases. A quick note to the dev mailing list indicating that you're investigating a potential issue would be sufficient and valuable. If I can't confirm the issue, what should I share? Our failed tests happen every day due to different problems, do I need to share them with the community every day? Penghui On Thu, Feb 17, 2022 at 3:41 AM Michael Marshall wrote: > I agree that it can be non-trivial to determine if an issue is an > internal test or an upstream issue. > > My main point is that we should prefer early communication of > potential issues that affect pending releases and accept that the > issue might prove to be nothing more than a short delay in the release > cycle. The alternative is that a release manager completes the build > process, which takes several hours, then discovers a new candidate is > needed. There is still an inherent race condition for release creation > and validation, but early communication (and collaboration) might help > us avoid some of the cases where candidates are rejected immediately. > > For what it's worth, I don't think you need to verify your test > correctness 100% before sharing potential issues that affect pending > releases. A quick note to the dev mailing list indicating that you're > investigating a potential issue would be sufficient and valuable. > > Thanks, > Michael > > > > > > On Tue, Feb 15, 2022 at 6:30 PM PengHui Li wrote: > > > > > Just a friendly reminder that we are an open source project and other > > community members who may have different internal testing may find better > > fixes. > > > > Sorry, dave. I know we are an open source project. > > We found a problem in our internal test, we need to confirm it is not > > the test issue right? > > > > Do you suggest that we discuss this in the email when we find our > internal > > test errors? > > The community has no responsibility to deal with our internal testing > bugs. > > > > On Wed, Feb 16, 2022 at 8:04 AM Dave Fisher wrote: > > > > > > > > > > > > On Feb 15, 2022, at 3:59 PM, PengHui Li wrote: > > > > > > > >> I wonder if we should be more willing to share suspicions on the > > > > mailing list when releases are actively being prepared. I would > > > > have appreciated the notice, and I could have possibly helped > > > > confirm the issue. On the other hand, I respect that it is hard to > > > > know when to share a theory and when to test it out a bit more. > > > > > > > > Yes, It's just that these issues are happening in our internal > testing > > > > and we need to 100% confirm the issue and then share it out, > > > > otherwise, We may confuse the contributors due to internal test > problems. > > > > > > Just a friendly reminder that we are an open source project and other > > > community members who may have different internal testing may find > better > > > fixes. > > > > > > > > > > > This kind of problem is often easy to fix, but difficult to > troubleshoot. > > > > So usually when you are 100% confirmed the issue, usually know how > to fix > > > > it. > > > > > > > >> I think these were not included because they were labeled after I > > > > cherry picked PRs on Friday. I will take a look and try to cherry > pick > > > > all of those before tagging the next release candidate > > > > > > > > Oh, I see. Thanks. I have checked > > > > https://github.com/apache/pulsar/commits/branch-2.8 > > > > LGTM. > > > > > > > > Regards, > > > > Penghui > > > > > > > > On Wed, Feb 16, 2022 at 1:21 AM Michael Marshall < > mmarsh...@apache.org> > > > > wrote: > > > > > > > >> Closing the vote with one -1 from Penghui. > > > >> > > > >>> We just confirmed yesterday that this is a breaking change, > > > >>> I was just suspicious before, so do not share the information > > > >>> to avoid confusing reviewers. For now, we confirmed it > > > >> > > > >> I
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Yes, but I think that the code freeze is only meaningful if it’s communicated in advance. The fact that it was included in the original PIP but never followed in the practice means it would be a last minute change. On Wed, Feb 16, 2022 at 2:37 PM Michael Marshall wrote: > When we discussed the code freeze in the community meeting on 2/3, I > was under the impression that it was a new development to our existing > release process. I subsequently learned it was already defined in > PIP 47. Even if we haven't been following this part of PIP 47, what > is the value in waiting until 2.11 to follow our already defined process? > While I agree it is helpful to provide guidance on when a version will > ship, > I think it is more important to give the community time to test a release, > even if that means we're a little late on our release schedule. So far, > we haven't even created a branch to begin testing. > > Note also that Sijie suggested using a feature freeze early on in this > thread. > > The 2.9.0 release is relevant here. It had 4 release candidates over 4 > weeks and the final result was broken. That indicates to me that tagging > an RC early does not guarantee an early release and that our current > process isn't optimal and likely needs adjustments. I do not think we > should wait to address these issues. I propose we start following > PIP 47's guidance on code freeze and release stabilization periods. > > > I don't think that changes the picture here. There are *always* last > > minute issues being discovered, and there is a call to be made on a > > case by case. The feature freeze will reduce the likelihood of > > introducing *more* issues by getting it from the master branch, but > > won't change a comma from issues that were already there. > > I thought you wanted to implement a code/feature freeze to allow for > more release stabilization. Can you clarify what you mean here? > > Thanks, > Michael > > > > > > > > On Wed, Feb 16, 2022 at 2:42 PM Matteo Merli > wrote: > > > > Michael, as we chatted in last weekly meeting (though not yet > > formalized), since we have never really done a feature freeze on the > > branch during paste releases, we should start from the next release, > > to give a decent preview of what to expect to developers in terms of > > dates. > > > > > While some may feel "behind" in getting out the 2.10 release, our > > > priority must be to give the community time to verify the stability of > > > the release. > > > > I don't think that changes the picture here. There are *always* last > > minute issues being discovered, and there is a call to be made on a > > case by case. The feature freeze will reduce the likelihood of > > introducing *more* issues by getting it from the master branch, but > > won't change a comma from issues that were already there. > > > > > > > > > > -- > > Matteo Merli > > > > > > On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall > wrote: > > > > > > > I will build the release and start the vote before next Monday(GMT+8) > > > > > > Penghui, is your current plan to create branch-2.10, create the > > > release artifacts, and start a vote on them all within a few days of > > > each other? > > > > > > > I'm doing my best to follow PIP 47, but when seeing a potential break > > > > change, I need to confirm it. > > > > After all the potential break changes have been confirmed and fixed, > I will > > > > start the vote thread. > > > > > > I think we should review our current release plan before we move > > > forward as proposed above. PIP 47 explicitly says that a month before > > > the release date, the release manager will cut branches [0]. We don't > > > yet have a `branch-2.10`. PIP 47 also defines a period of time for a > > > feature freeze and then a code freeze. We have not yet had either. > > > > > > I propose we create branch-2.10 now and simultaneously announce that > > > we are past the feature freeze period. Then, we can start the 2 week > > > period for bug fixes that precedes the code freeze, as PIP 47 > > > prescribes. Then, in two weeks, we can produce the first release > > > candidate (also in PIP 47). > > > > > > While some may feel "behind" in getting out the 2.10 release, our > > > priority must be to give the community time to verify the stability of > > > the release. > > > > > > Thanks, > > > Michael > > > > > > [0] > https://github.com/apache/pulsar/wiki/PIP-47%3A-Time-Based-Release-Plan > > > > > > > > > > > > > > > On Wed, Feb 16, 2022 at 9:09 AM PengHui Li wrote: > > > > > > > > Hi all > > > > > > > > Just put an update here. > > > > > > > > We have 2 PRs[1] https://github.com/apache/pulsar/pull/13376 and > > > > https://github.com/apache/pulsar/pull/13341 > > > > need to do the final verification, and you are also very welcome to > verify > > > > these 2 changes in your environment, cases. > > > > > > > > I will build the release and start the vote before next Monday(GMT+8) > > > > > > > > Regards > > > > Penghui > > > > > > >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Hi Micheal > Penghui, is your current plan to create branch-2.10, create the release artifacts, and start a vote on them all within a few days of each other? Yes, I will create branch-2.10 today. For starting the vote, we need to confirm these 2 PRs[1] will not introduce breaking changes. Very grateful if someone can also help verify them. [1] https://github.com/apache/pulsar/pull/13376, https://github.com/apache/pulsar/pull/13341 Thanks, Penghui On Thu, Feb 17, 2022 at 8:59 AM Matteo Merli wrote: > Yes, but I think that the code freeze is only meaningful if it’s > communicated in advance. > > The fact that it was included in the original PIP but never followed in the > practice means it would be a last minute change. > > On Wed, Feb 16, 2022 at 2:37 PM Michael Marshall > wrote: > > > When we discussed the code freeze in the community meeting on 2/3, I > > was under the impression that it was a new development to our existing > > release process. I subsequently learned it was already defined in > > PIP 47. Even if we haven't been following this part of PIP 47, what > > is the value in waiting until 2.11 to follow our already defined process? > > While I agree it is helpful to provide guidance on when a version will > > ship, > > I think it is more important to give the community time to test a > release, > > even if that means we're a little late on our release schedule. So far, > > we haven't even created a branch to begin testing. > > > > Note also that Sijie suggested using a feature freeze early on in this > > thread. > > > > The 2.9.0 release is relevant here. It had 4 release candidates over 4 > > weeks and the final result was broken. That indicates to me that tagging > > an RC early does not guarantee an early release and that our current > > process isn't optimal and likely needs adjustments. I do not think we > > should wait to address these issues. I propose we start following > > PIP 47's guidance on code freeze and release stabilization periods. > > > > > I don't think that changes the picture here. There are *always* last > > > minute issues being discovered, and there is a call to be made on a > > > case by case. The feature freeze will reduce the likelihood of > > > introducing *more* issues by getting it from the master branch, but > > > won't change a comma from issues that were already there. > > > > I thought you wanted to implement a code/feature freeze to allow for > > more release stabilization. Can you clarify what you mean here? > > > > Thanks, > > Michael > > > > > > > > > > > > > > > > On Wed, Feb 16, 2022 at 2:42 PM Matteo Merli > > wrote: > > > > > > Michael, as we chatted in last weekly meeting (though not yet > > > formalized), since we have never really done a feature freeze on the > > > branch during paste releases, we should start from the next release, > > > to give a decent preview of what to expect to developers in terms of > > > dates. > > > > > > > While some may feel "behind" in getting out the 2.10 release, our > > > > priority must be to give the community time to verify the stability > of > > > > the release. > > > > > > I don't think that changes the picture here. There are *always* last > > > minute issues being discovered, and there is a call to be made on a > > > case by case. The feature freeze will reduce the likelihood of > > > introducing *more* issues by getting it from the master branch, but > > > won't change a comma from issues that were already there. > > > > > > > > > > > > > > > -- > > > Matteo Merli > > > > > > > > > On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall < > mmarsh...@apache.org> > > wrote: > > > > > > > > > I will build the release and start the vote before next > Monday(GMT+8) > > > > > > > > Penghui, is your current plan to create branch-2.10, create the > > > > release artifacts, and start a vote on them all within a few days of > > > > each other? > > > > > > > > > I'm doing my best to follow PIP 47, but when seeing a potential > break > > > > > change, I need to confirm it. > > > > > After all the potential break changes have been confirmed and > fixed, > > I will > > > > > start the vote thread. > > > > > > > > I think we should review our current release plan before we move > > > > forward as proposed above. PIP 47 explicitly says that a month before > > > > the release date, the release manager will cut branches [0]. We don't > > > > yet have a `branch-2.10`. PIP 47 also defines a period of time for a > > > > feature freeze and then a code freeze. We have not yet had either. > > > > > > > > I propose we create branch-2.10 now and simultaneously announce that > > > > we are past the feature freeze period. Then, we can start the 2 week > > > > period for bug fixes that precedes the code freeze, as PIP 47 > > > > prescribes. Then, in two weeks, we can produce the first release > > > > candidate (also in PIP 47). > > > > > > > > While some may feel "behind" in getting out the 2.10 release, our > > > > priority must be t
Re: [VOTE] Pulsar Client Go Release 0.8.0 Candidate 1
+1 - Checked the signatures - Run produce and consume on a standalone Penghui On Thu, Feb 17, 2022 at 12:45 AM Matteo Merli wrote: > +1 > > Checked signatures, compiled and run perf tool. > > > > -- > Matteo Merli > > > On Tue, Feb 15, 2022 at 7:28 PM r...@apache.org > wrote: > > > > Hi everyone, > > Please review and vote on the release candidate #1 for the version > > 0.8.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This is the first release candidate for Apache Pulsar Go client, version > 0.8.0. > > > > It fixes the following issues: > > https://github.com/apache/pulsar-client-go/milestone/9?closed=1 > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Please download these packages and review this release candidate: > > - Review release notes > > - Download the source package (verify shasum, and asc) and follow the > > README.md to build and run the pulsar-client-go. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source file: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.8.0-candidate-1/ > > > > The tag to be voted upon: > > v0.8.0 > > https://github.com/apache/pulsar-client-go/tree/v0.8.0-candidate-1 > > > > SHA-512 checksums: > > > 0015cc735af13139a564f0fd57c58af753579953e3c12f08cb62d8e9801cdc2964c60392ed9a98dd54845be7f7b1a353bd49cecb03360b4cb691904f8447769c > > apache-pulsar-client-go-0.8.0-src.tar.gz >
Re: [VOTE] Pulsar Release 2.8.3 Candidate 2
+1 (binding) - Checked the signature - Start the standalone - Publish and consume messages - Check the Cassandra connector - Check the stateful function Thanks for the great work, Penghui On Wed, Feb 16, 2022 at 12:35 PM Michael Marshall wrote: > This is the second release candidate for Apache Pulsar, version 2.8.3. > > It fixes the following issues: > https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/ > > SHA-512 checksums: > > > aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090 > ./apache-pulsar-2.8.3-bin.tar.gz > > 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121 > ./apache-pulsar-2.8.3-src.tar.gz > > Unofficial Docker images: > michaelmarshall/pulsar:2.8.3-rc2 > michaelmarshall/pulsar-all:2.8.3-rc2 > michaelmarshall/pulsar-standalone:2.8.3-rc2 > michaelmarshall/pulsar-grafana:2.8.3-rc2 > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1140/ > > The tag to be voted upon: > v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b) > https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build > and run the Pulsar standalone service. >
Re: [DISCUSS] PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
Hi Lari, thanks for the proposal, though I don't think it would be good to change this default value. Pulsar HTTP API is an administrative service and it is not on the critical data path. If all the threads in the pool are busy, it is good to apply back pressure and slow down the incoming new requests. In the past, we had several cases of deadlocks involving HTTP Jetty threads. In all these cases, the root cause has been a broker that in response to an HTTP request is spawning HTTP requests to other brokers, and some of these requests are going to itself. These kinds of deadlock are unrelated to the number of HTTP threads in the pool and must be dealt with by making the underlying operation asynchronous. Finally each user is free to change the default to something that suits more, though I really don't think it makes for us to default to 200 threads as: 1. It will not improve the Pulsar performance 2. It will not magically solve any deadlock issue 3. It will just consume much more resources for everyone Replying in-line to some of the points. > > Additional context > > - Examples of previous issues where Jetty threads have been occupied > and caused problems: #13666 #4756 #10619 All these examples are cases of what I have described above: an HTTP handler triggering a call to the same HTTP server resulting in a deadlock. Having had more threads wouldn't have fixed them. > - Mailing list thread about “make async” changes: >https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l The rationale for these changes has not much to do with Jetty but rather to make sure that we are not mixing blocking and non-blocking operations when handling these requests, which is prone to cause deadlocks in broker's internal threads (irrespective of any Jetty thread pool size). > Q: What’s the reason of setting the default value to 200? If the node just > have one core, what will happen? > > These are threads. Jetty defaults to 200 maximum threads, to prevent > thread pool starvation. This is recommended when using blocking Servlet > API. The problem is that Pulsar uses the blocking servlet API and > doesn’t have a sufficient number of threads which are needed and > recommended. We need to keep in mind that typical usage of Jetty is as a Servlet container, dedicated to that. You have servlets doing disk IO reads and blocking calls on JDBC drivers. Pulsar usage is *very* different from that. Admin API is a tiny portion of what a broker does. Dedicating 200 threads to that would be a massive waste of resources (CPU & memory). > The value 200 doesn’t mean that there will be 200 threads to start with. > This is the maximum size for the thread pool. When the value is more > than 8, Jetty will start with 8 initial threads and add more threads to > the pool when all threads are occupied. > > Q: Do we need to take the number of system cores into consideration for the > maximum threads of the thread pool? > > No. Jetty is different from Netty in this aspect. In Netty, everything > should be asynchronous and “thou shall never block”. In Jetty, the > maximum number of threads for the thread pool should be set to 50-500 > threads and blocking operations are fine. I disagree. In a container configured with 0.5 CPU cores, it would be very bad to run 200 threads, and it will have no other advantage whatsoever. That is the reason why the default thread pool size is pegged to the number of cores. > The recommendation for the thread pool is explained in Jetty > documentation > https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool > > Thread Pool > Configure with goal of limiting memory usage maximum > available. Typically this is >50 and <500 The recommendations for Jetty are relative to typical Jetty workloads, something the Pulsar broker is definitely not.
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Hi all, Put an update here, I have created branch-2.10[1] [1]https://github.com/apache/pulsar/tree/branch-2.10 On Thu, Feb 17, 2022 at 9:23 AM PengHui Li wrote: > Hi Micheal > > > Penghui, is your current plan to create branch-2.10, create the > release artifacts, and start a vote on them all within a few days of > each other? > > Yes, I will create branch-2.10 today. > > For starting the vote, we need to confirm these 2 PRs[1] will not introduce > breaking changes. Very grateful if someone can also help verify them. > > > [1] https://github.com/apache/pulsar/pull/13376, > https://github.com/apache/pulsar/pull/13341 > > Thanks, > Penghui > > On Thu, Feb 17, 2022 at 8:59 AM Matteo Merli > wrote: > >> Yes, but I think that the code freeze is only meaningful if it’s >> communicated in advance. >> >> The fact that it was included in the original PIP but never followed in >> the >> practice means it would be a last minute change. >> >> On Wed, Feb 16, 2022 at 2:37 PM Michael Marshall >> wrote: >> >> > When we discussed the code freeze in the community meeting on 2/3, I >> > was under the impression that it was a new development to our existing >> > release process. I subsequently learned it was already defined in >> > PIP 47. Even if we haven't been following this part of PIP 47, what >> > is the value in waiting until 2.11 to follow our already defined >> process? >> > While I agree it is helpful to provide guidance on when a version will >> > ship, >> > I think it is more important to give the community time to test a >> release, >> > even if that means we're a little late on our release schedule. So far, >> > we haven't even created a branch to begin testing. >> > >> > Note also that Sijie suggested using a feature freeze early on in this >> > thread. >> > >> > The 2.9.0 release is relevant here. It had 4 release candidates over 4 >> > weeks and the final result was broken. That indicates to me that tagging >> > an RC early does not guarantee an early release and that our current >> > process isn't optimal and likely needs adjustments. I do not think we >> > should wait to address these issues. I propose we start following >> > PIP 47's guidance on code freeze and release stabilization periods. >> > >> > > I don't think that changes the picture here. There are *always* last >> > > minute issues being discovered, and there is a call to be made on a >> > > case by case. The feature freeze will reduce the likelihood of >> > > introducing *more* issues by getting it from the master branch, but >> > > won't change a comma from issues that were already there. >> > >> > I thought you wanted to implement a code/feature freeze to allow for >> > more release stabilization. Can you clarify what you mean here? >> > >> > Thanks, >> > Michael >> > >> > >> > >> > >> > >> > >> > >> > On Wed, Feb 16, 2022 at 2:42 PM Matteo Merli >> > wrote: >> > > >> > > Michael, as we chatted in last weekly meeting (though not yet >> > > formalized), since we have never really done a feature freeze on the >> > > branch during paste releases, we should start from the next release, >> > > to give a decent preview of what to expect to developers in terms of >> > > dates. >> > > >> > > > While some may feel "behind" in getting out the 2.10 release, our >> > > > priority must be to give the community time to verify the stability >> of >> > > > the release. >> > > >> > > I don't think that changes the picture here. There are *always* last >> > > minute issues being discovered, and there is a call to be made on a >> > > case by case. The feature freeze will reduce the likelihood of >> > > introducing *more* issues by getting it from the master branch, but >> > > won't change a comma from issues that were already there. >> > > >> > > >> > > >> > > >> > > -- >> > > Matteo Merli >> > > >> > > >> > > On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall < >> mmarsh...@apache.org> >> > wrote: >> > > > >> > > > > I will build the release and start the vote before next >> Monday(GMT+8) >> > > > >> > > > Penghui, is your current plan to create branch-2.10, create the >> > > > release artifacts, and start a vote on them all within a few days of >> > > > each other? >> > > > >> > > > > I'm doing my best to follow PIP 47, but when seeing a potential >> break >> > > > > change, I need to confirm it. >> > > > > After all the potential break changes have been confirmed and >> fixed, >> > I will >> > > > > start the vote thread. >> > > > >> > > > I think we should review our current release plan before we move >> > > > forward as proposed above. PIP 47 explicitly says that a month >> before >> > > > the release date, the release manager will cut branches [0]. We >> don't >> > > > yet have a `branch-2.10`. PIP 47 also defines a period of time for a >> > > > feature freeze and then a code freeze. We have not yet had either. >> > > > >> > > > I propose we create branch-2.10 now and simultaneously announce that >> > > > we are past the feature freeze period. Th
Re: [VOTE] Pulsar Release 2.8.3 Candidate 2
Hi Michael, Please check your key ``` gpg: assuming signed data in 'apache-pulsar-2.8.3-bin.tar.gz' gpg: Signature made Wed Feb 16 04:49:42 2022 CST gpg:using RSA key BD4291E509D771B79E7BD1F5C5724B3F5588C4EB gpg: Good signature from "Michael Marshall " [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. ``` I think it should be an issue that the key does not been signed? I'm not sure. Thanks, Penghui On Thu, Feb 17, 2022 at 9:56 AM PengHui Li wrote: > +1 (binding) > > - Checked the signature > - Start the standalone > - Publish and consume messages > - Check the Cassandra connector > - Check the stateful function > > Thanks for the great work, > Penghui > > On Wed, Feb 16, 2022 at 12:35 PM Michael Marshall > wrote: > >> This is the second release candidate for Apache Pulsar, version 2.8.3. >> >> It fixes the following issues: >> https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2 >> >> *** Please download, test and vote on this release. This vote will stay >> open >> for at least 72 hours *** >> >> Note that we are voting upon the source (tag), binaries are provided for >> convenience. >> >> Source and binary files: >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/ >> >> SHA-512 checksums: >> >> >> aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090 >> ./apache-pulsar-2.8.3-bin.tar.gz >> >> 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121 >> ./apache-pulsar-2.8.3-src.tar.gz >> >> Unofficial Docker images: >> michaelmarshall/pulsar:2.8.3-rc2 >> michaelmarshall/pulsar-all:2.8.3-rc2 >> michaelmarshall/pulsar-standalone:2.8.3-rc2 >> michaelmarshall/pulsar-grafana:2.8.3-rc2 >> >> Maven staging repo: >> https://repository.apache.org/content/repositories/orgapachepulsar-1140/ >> >> The tag to be voted upon: >> v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b) >> https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2 >> >> Pulsar's KEYS file containing PGP keys we use to sign the release: >> https://dist.apache.org/repos/dist/dev/pulsar/KEYS >> >> Please download the source package, and follow the README to build >> and run the Pulsar standalone service. >> >
Re: [VOTE] Pulsar Release 2.9.2 Candidate 3
+1 (binding) - Checked the signature - Build from the source - Start the standalone - Publish and consume messages - Check the Cassandra connector - Check the stateful function Thanks for the great work, Penghui On Thu, Feb 17, 2022 at 2:26 AM Ran Gao wrote: > This is the third release candidate for Apache Pulsar, version 2.9.2. > > *** Please download, test, and vote on this release. This vote will stay > open for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.2-candidate-3/ > > SHA-512 checksums: > > > 21ab21df057dc181d02806e67a8d85b861fa60a8413a8e4058ffcb738d9ef81d8797c4e7249d50cf2a23c7437cb2880f4240d010a983eda33a2071f5fddf6b01 > apache-pulsar-2.9.2-bin.tar.gz > > f927d77aba32d1112cd009a1d91a8fca27a49fcb20ea215a5792dc16e2b04bd822238c3f29dd81a7853e20062a0454e2abe608626b24e4e25ff06915b7a04785 > apache-pulsar-2.9.2-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1141/ > > The tag to be voted upon: > v2.9.2-candidate-3 (5615809ffd68599ae7fec1be0679ee83171f42dc) > https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-3 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build and run > the Pulsar standalone service. >
[GitHub] [pulsar-site] Anonymitaet merged pull request #1: [PIP-87][Website] Publish website from asf-site-next branch
Anonymitaet merged pull request #1: URL: https://github.com/apache/pulsar-site/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] Pulsar Release 2.8.3 Candidate 2
While it’s preferred that KEYS are signed it’s not required. The last Apache key signing party was at Apachecon Las Vegas in 2019. Here’s looking forward to the next one. Bring your government ids. Regards, Dave Sent from my iPhone > On Feb 16, 2022, at 6:11 PM, PengHui Li wrote: > > Hi Michael, > > Please check your key > > ``` > gpg: assuming signed data in 'apache-pulsar-2.8.3-bin.tar.gz' > gpg: Signature made Wed Feb 16 04:49:42 2022 CST > gpg:using RSA key BD4291E509D771B79E7BD1F5C5724B3F5588C4EB > gpg: Good signature from "Michael Marshall " [unknown] > gpg: WARNING: This key is not certified with a trusted signature! > gpg: There is no indication that the signature belongs to the > owner. > ``` > > I think it should be an issue that the key does not been signed? I'm not > sure. > > Thanks, > Penghui > >> On Thu, Feb 17, 2022 at 9:56 AM PengHui Li wrote: >> >> +1 (binding) >> >> - Checked the signature >> - Start the standalone >> - Publish and consume messages >> - Check the Cassandra connector >> - Check the stateful function >> >> Thanks for the great work, >> Penghui >> >> On Wed, Feb 16, 2022 at 12:35 PM Michael Marshall >> wrote: >> >>> This is the second release candidate for Apache Pulsar, version 2.8.3. >>> >>> It fixes the following issues: >>> https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2 >>> >>> *** Please download, test and vote on this release. This vote will stay >>> open >>> for at least 72 hours *** >>> >>> Note that we are voting upon the source (tag), binaries are provided for >>> convenience. >>> >>> Source and binary files: >>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/ >>> >>> SHA-512 checksums: >>> >>> >>> aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090 >>> ./apache-pulsar-2.8.3-bin.tar.gz >>> >>> 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121 >>> ./apache-pulsar-2.8.3-src.tar.gz >>> >>> Unofficial Docker images: >>> michaelmarshall/pulsar:2.8.3-rc2 >>> michaelmarshall/pulsar-all:2.8.3-rc2 >>> michaelmarshall/pulsar-standalone:2.8.3-rc2 >>> michaelmarshall/pulsar-grafana:2.8.3-rc2 >>> >>> Maven staging repo: >>> https://repository.apache.org/content/repositories/orgapachepulsar-1140/ >>> >>> The tag to be voted upon: >>> v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b) >>> https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2 >>> >>> Pulsar's KEYS file containing PGP keys we use to sign the release: >>> https://dist.apache.org/repos/dist/dev/pulsar/KEYS >>> >>> Please download the source package, and follow the README to build >>> and run the Pulsar standalone service. >>> >>
Re: [DISCUSS] PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
Hi Lari, Thanks for raising a proposal and starting the discussion thread here. Increasing the thread does not solve the underlying problem. but also brings additional resource consumption. Thanks, Matteo provided a great explanation, and, I totally agree with his explanation. What should we try to optimize thread usage, reduce block calls, rather than going in the opposite direction. For the current REST API implementation, I saw 2 main problems 1. The API defined we are using the Jetty async mode, but is actually a blocking call. I think 100% this is bad code, confuse contributors, and bring potential risks 2. We don't know the exact return type from the interface definition [1] until reading the implementation code here[2] So, I think to change the current implementation to purely async and avoid passing the `AsyncResponse` to the implementation is the right way to improve the REST API and fix the potential problems. [1] https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java#L1073 [2] https://github.com/apache/pulsar/blob/df9a12dc90316e0d5c1d9effb416ec83fbcf9b5e/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java#L1259 Thanks, Penghui On Thu, Feb 17, 2022 at 10:00 AM Matteo Merli wrote: > Hi Lari, > > thanks for the proposal, though I don't think it would be good to > change this default value. > > Pulsar HTTP API is an administrative service and it is not on the > critical data path. > If all the threads in the pool are busy, it is good to apply back > pressure and slow down the incoming new requests. > > In the past, we had several cases of deadlocks involving HTTP Jetty > threads. In all these cases, the root cause has been a broker that in > response to an HTTP request is spawning HTTP requests to other > brokers, and some of these requests are going to itself. > > These kinds of deadlock are unrelated to the number of HTTP threads in > the pool and must be dealt with by making the underlying operation > asynchronous. > > Finally each user is free to change the default to something that > suits more, though I really don't think it makes for us to default to > 200 threads as: > 1. It will not improve the Pulsar performance > 2. It will not magically solve any deadlock issue > 3. It will just consume much more resources for everyone > > > Replying in-line to some of the points. > > > > > Additional context > > > > - Examples of previous issues where Jetty threads have been occupied > > and caused problems: #13666 #4756 #10619 > > All these examples are cases of what I have described above: an HTTP > handler triggering a call to the same HTTP server resulting in a > deadlock. > > Having had more threads wouldn't have fixed them. > > > - Mailing list thread about “make async” changes: > >https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l > > The rationale for these changes has not much to do with Jetty but > rather to make sure that we are not mixing blocking and non-blocking > operations when handling these requests, which is prone to cause > deadlocks in broker's internal threads (irrespective of any Jetty > thread pool size). > > > Q: What’s the reason of setting the default value to 200? If the node > just have one core, what will happen? > > > > These are threads. Jetty defaults to 200 maximum threads, to prevent > > thread pool starvation. This is recommended when using blocking Servlet > > API. The problem is that Pulsar uses the blocking servlet API and > > doesn’t have a sufficient number of threads which are needed and > > recommended. > > We need to keep in mind that typical usage of Jetty is as a Servlet > container, dedicated to that. You have servlets doing disk IO reads > and blocking calls on JDBC drivers. > > Pulsar usage is *very* different from that. Admin API is a tiny > portion of what a broker does. Dedicating 200 threads to that would be > a massive waste of resources (CPU & memory). > > > The value 200 doesn’t mean that there will be 200 threads to start with. > > This is the maximum size for the thread pool. When the value is more > > than 8, Jetty will start with 8 initial threads and add more threads to > > the pool when all threads are occupied. > > > > Q: Do we need to take the number of system cores into consideration for > the maximum threads of the thread pool? > > > > No. Jetty is different from Netty in this aspect. In Netty, everything > > should be asynchronous and “thou shall never block”. In Jetty, the > > maximum number of threads for the thread pool should be set to 50-500 > > threads and blocking operations are fine. > > I disagree. In a container configured with 0.5 CPU cores, it would be > very bad to run 200 threads, and it will have no other advantage > whatsoever. > > That is the reason why the default thread pool size is pegged to the > number of cores. > > > The recommendation for the thread pool is expla
Re: [VOTE] Pulsar Release 2.8.3 Candidate 2
Thanks for the explanation Dave. Best, Penghui On Thu, Feb 17, 2022 at 10:37 AM Dave Fisher wrote: > While it’s preferred that KEYS are signed it’s not required. The last > Apache key signing party was at Apachecon Las Vegas in 2019. Here’s looking > forward to the next one. Bring your government ids. > > Regards, > Dave > > Sent from my iPhone > > > On Feb 16, 2022, at 6:11 PM, PengHui Li wrote: > > > > Hi Michael, > > > > Please check your key > > > > ``` > > gpg: assuming signed data in 'apache-pulsar-2.8.3-bin.tar.gz' > > gpg: Signature made Wed Feb 16 04:49:42 2022 CST > > gpg:using RSA key > BD4291E509D771B79E7BD1F5C5724B3F5588C4EB > > gpg: Good signature from "Michael Marshall " > [unknown] > > gpg: WARNING: This key is not certified with a trusted signature! > > gpg: There is no indication that the signature belongs to the > > owner. > > ``` > > > > I think it should be an issue that the key does not been signed? I'm not > > sure. > > > > Thanks, > > Penghui > > > >> On Thu, Feb 17, 2022 at 9:56 AM PengHui Li wrote: > >> > >> +1 (binding) > >> > >> - Checked the signature > >> - Start the standalone > >> - Publish and consume messages > >> - Check the Cassandra connector > >> - Check the stateful function > >> > >> Thanks for the great work, > >> Penghui > >> > >> On Wed, Feb 16, 2022 at 12:35 PM Michael Marshall > > >> wrote: > >> > >>> This is the second release candidate for Apache Pulsar, version 2.8.3. > >>> > >>> It fixes the following issues: > >>> https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2 > >>> > >>> *** Please download, test and vote on this release. This vote will stay > >>> open > >>> for at least 72 hours *** > >>> > >>> Note that we are voting upon the source (tag), binaries are provided > for > >>> convenience. > >>> > >>> Source and binary files: > >>> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/ > >>> > >>> SHA-512 checksums: > >>> > >>> > >>> > aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090 > >>> ./apache-pulsar-2.8.3-bin.tar.gz > >>> > >>> > 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121 > >>> ./apache-pulsar-2.8.3-src.tar.gz > >>> > >>> Unofficial Docker images: > >>> michaelmarshall/pulsar:2.8.3-rc2 > >>> michaelmarshall/pulsar-all:2.8.3-rc2 > >>> michaelmarshall/pulsar-standalone:2.8.3-rc2 > >>> michaelmarshall/pulsar-grafana:2.8.3-rc2 > >>> > >>> Maven staging repo: > >>> > https://repository.apache.org/content/repositories/orgapachepulsar-1140/ > >>> > >>> The tag to be voted upon: > >>> v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b) > >>> https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2 > >>> > >>> Pulsar's KEYS file containing PGP keys we use to sign the release: > >>> https://dist.apache.org/repos/dist/dev/pulsar/KEYS > >>> > >>> Please download the source package, and follow the README to build > >>> and run the Pulsar standalone service. > >>> > >> > >
New Pulsar website goes live!
Hi Pulsarers, As you may notice, the new Pulsar website goes live! https://pulsar.apache.org/ We hope you enjoy the new feature and the fresh appearance! CHANGE: documentation main page URL is changed from https://pulsar.apache.org/docs/en/next/standalone/ to https://pulsar.apache.org/docs/next/. It does not affect user experience if you read documentation navigated from the Pulsar main page. Besides, there are some known issues (eg. localized pages and some CLI pages are not available). We're fixing them. Feel free to join us or make comments, thanks! Special thanks go to @urfreespace and @dave2wave to make this happen! Anonymitaet
Re: New Pulsar website goes live!
Great news! Thanks for the great work @Yu, @@urfreespace, @dave2wave. Penghui On Thu, Feb 17, 2022 at 11:15 AM Yu wrote: > Hi Pulsarers, > > As you may notice, the new Pulsar website goes live! > https://pulsar.apache.org/ > > We hope you enjoy the new feature and the fresh appearance! > > CHANGE: documentation main page URL is changed from > https://pulsar.apache.org/docs/en/next/standalone/ to > https://pulsar.apache.org/docs/next/. > > It does not affect user experience if you read documentation navigated > from the Pulsar main page. > > Besides, there are some known issues (eg. localized pages and some CLI > pages are not available). We're fixing them. Feel free to join us or make > comments, thanks! > > Special thanks go to @urfreespace and @dave2wave to make this happen! > > > Anonymitaet >
Re: New Pulsar website goes live!
I just go through the new website, looks like the CLI's document always return code 404 I have created an issue for tracking the problem https://github.com/apache/pulsar/issues/14332 Thanks, Penghui On Thu, Feb 17, 2022 at 11:32 AM PengHui Li wrote: > Great news! > > Thanks for the great work @Yu, @@urfreespace, @dave2wave. > > Penghui > > On Thu, Feb 17, 2022 at 11:15 AM Yu wrote: > >> Hi Pulsarers, >> >> As you may notice, the new Pulsar website goes live! >> https://pulsar.apache.org/ >> >> We hope you enjoy the new feature and the fresh appearance! >> >> CHANGE: documentation main page URL is changed from >> https://pulsar.apache.org/docs/en/next/standalone/ to >> https://pulsar.apache.org/docs/next/. >> >> It does not affect user experience if you read documentation navigated >> from the Pulsar main page. >> >> Besides, there are some known issues (eg. localized pages and some CLI >> pages are not available). We're fixing them. Feel free to join us or make >> comments, thanks! >> >> Special thanks go to @urfreespace and @dave2wave to make this happen! >> >> >> Anonymitaet >> >
Re: New Pulsar website goes live!
Great work! Any plans for multi-language translation support? There were some languages supported, such as Chinese, Japanese, French and more, but now cannot find the button in the new website. Best, Dianjin Wang On Thu, Feb 17, 2022 at 11:15 AM Yu wrote: > Hi Pulsarers, > > As you may notice, the new Pulsar website goes live! > https://pulsar.apache.org/ > > We hope you enjoy the new feature and the fresh appearance! > > CHANGE: documentation main page URL is changed from > https://pulsar.apache.org/docs/en/next/standalone/ to > https://pulsar.apache.org/docs/next/. > > It does not affect user experience if you read documentation navigated > from the Pulsar main page. > > Besides, there are some known issues (eg. localized pages and some CLI > pages are not available). We're fixing them. Feel free to join us or make > comments, thanks! > > Special thanks go to @urfreespace and @dave2wave to make this happen! > > > Anonymitaet >
Re: [VOTE] Pulsar Client Go Release 0.8.0 Candidate 1
+1 Checked the signatures, complied and ran the pular perf tool (produce and consume) against a standalone cluster. On Wed, Feb 16, 2022 at 5:29 PM PengHui Li wrote: > > +1 > > - Checked the signatures > - Run produce and consume on a standalone > > Penghui > > On Thu, Feb 17, 2022 at 12:45 AM Matteo Merli > wrote: > > > +1 > > > > Checked signatures, compiled and run perf tool. > > > > > > > > -- > > Matteo Merli > > > > > > On Tue, Feb 15, 2022 at 7:28 PM r...@apache.org > > wrote: > > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version > > > 0.8.0, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This is the first release candidate for Apache Pulsar Go client, version > > 0.8.0. > > > > > > It fixes the following issues: > > > https://github.com/apache/pulsar-client-go/milestone/9?closed=1 > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > > release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Please download these packages and review this release candidate: > > > - Review release notes > > > - Download the source package (verify shasum, and asc) and follow the > > > README.md to build and run the pulsar-client-go. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Source file: > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.8.0-candidate-1/ > > > > > > The tag to be voted upon: > > > v0.8.0 > > > https://github.com/apache/pulsar-client-go/tree/v0.8.0-candidate-1 > > > > > > SHA-512 checksums: > > > > > 0015cc735af13139a564f0fd57c58af753579953e3c12f08cb62d8e9801cdc2964c60392ed9a98dd54845be7f7b1a353bd49cecb03360b4cb691904f8447769c > > > apache-pulsar-client-go-0.8.0-src.tar.gz > >
Re: New Pulsar website goes live!
Thanks for your quick feedback, Penghui and Dianjin. As mentioned in the last email, "*...**there are some known issues (eg. localized pages and some CLI pages are not available). We're fixing them* *...* " We'll keep you updated once we resolve them. On Thu, Feb 17, 2022 at 11:52 AM Dianjin Wang wrote: > Great work! > > Any plans for multi-language translation support? There were some languages > supported, such as Chinese, Japanese, French and more, but now cannot find > the button in the new website. > > Best, > Dianjin Wang > > > On Thu, Feb 17, 2022 at 11:15 AM Yu wrote: > > > Hi Pulsarers, > > > > As you may notice, the new Pulsar website goes live! > > https://pulsar.apache.org/ > > > > We hope you enjoy the new feature and the fresh appearance! > > > > CHANGE: documentation main page URL is changed from > > https://pulsar.apache.org/docs/en/next/standalone/ to > > https://pulsar.apache.org/docs/next/. > > > > It does not affect user experience if you read documentation navigated > > from the Pulsar main page. > > > > Besides, there are some known issues (eg. localized pages and some CLI > > pages are not available). We're fixing them. Feel free to join us or make > > comments, thanks! > > > > Special thanks go to @urfreespace and @dave2wave to make this happen! > > > > > > Anonymitaet > > >
Re: [VOTE] Pulsar Release 2.8.3 Candidate 1
Thanks for your reply, Penghui. I saw your note on that thread but didn't realize it affected 2.8.3. I absolutely did not mean to imply that you don't want to share here. I appreciate your many contributions to the community and the mailing list, and I look forward to future collaboration. Thanks, Michael On Wed, Feb 16, 2022 at 6:58 PM PengHui Li wrote: > > > I agree that it can be non-trivial to determine if an issue is an > internal test or an upstream issue. > > My main point is that we should prefer early communication of > potential issues that affect pending releases and accept that the > issue might prove to be nothing more than a short delay in the release > cycle. The alternative is that a release manager completes the build > process, which takes several hours, then discovers a new candidate is > needed. There is still an inherent race condition for release creation > and validation, but early communication (and collaboration) might help > us avoid some of the cases where candidates are rejected immediately. > > I have explained here > https://lists.apache.org/thread/m5so1scfok44hnd74436w100smn2vbh3 > at 11/02/2022 > Have you seen it? As I said before, I forget to update this thread, > I have declared at the first time that I respond to this thread. > it's not that I don't want to share to this thread. > I can't accept dave and you thinking I don't want to share here. > it's completely different > > > For what it's worth, I don't think you need to verify your test > correctness 100% before sharing potential issues that affect pending > releases. A quick note to the dev mailing list indicating that you're > investigating a potential issue would be sufficient and valuable. > > If I can't confirm the issue, what should I share? > Our failed tests happen every day due to different problems, > do I need to share them with the community every day? > > Penghui > > On Thu, Feb 17, 2022 at 3:41 AM Michael Marshall > wrote: > > > I agree that it can be non-trivial to determine if an issue is an > > internal test or an upstream issue. > > > > My main point is that we should prefer early communication of > > potential issues that affect pending releases and accept that the > > issue might prove to be nothing more than a short delay in the release > > cycle. The alternative is that a release manager completes the build > > process, which takes several hours, then discovers a new candidate is > > needed. There is still an inherent race condition for release creation > > and validation, but early communication (and collaboration) might help > > us avoid some of the cases where candidates are rejected immediately. > > > > For what it's worth, I don't think you need to verify your test > > correctness 100% before sharing potential issues that affect pending > > releases. A quick note to the dev mailing list indicating that you're > > investigating a potential issue would be sufficient and valuable. > > > > Thanks, > > Michael > > > > > > > > > > > > On Tue, Feb 15, 2022 at 6:30 PM PengHui Li wrote: > > > > > > > Just a friendly reminder that we are an open source project and other > > > community members who may have different internal testing may find better > > > fixes. > > > > > > Sorry, dave. I know we are an open source project. > > > We found a problem in our internal test, we need to confirm it is not > > > the test issue right? > > > > > > Do you suggest that we discuss this in the email when we find our > > internal > > > test errors? > > > The community has no responsibility to deal with our internal testing > > bugs. > > > > > > On Wed, Feb 16, 2022 at 8:04 AM Dave Fisher wrote: > > > > > > > > > > > > > > > > On Feb 15, 2022, at 3:59 PM, PengHui Li wrote: > > > > > > > > > >> I wonder if we should be more willing to share suspicions on the > > > > > mailing list when releases are actively being prepared. I would > > > > > have appreciated the notice, and I could have possibly helped > > > > > confirm the issue. On the other hand, I respect that it is hard to > > > > > know when to share a theory and when to test it out a bit more. > > > > > > > > > > Yes, It's just that these issues are happening in our internal > > testing > > > > > and we need to 100% confirm the issue and then share it out, > > > > > otherwise, We may confuse the contributors due to internal test > > problems. > > > > > > > > Just a friendly reminder that we are an open source project and other > > > > community members who may have different internal testing may find > > better > > > > fixes. > > > > > > > > > > > > > > This kind of problem is often easy to fix, but difficult to > > troubleshoot. > > > > > So usually when you are 100% confirmed the issue, usually know how > > to fix > > > > > it. > > > > > > > > > >> I think these were not included because they were labeled after I > > > > > cherry picked PRs on Friday. I will take a look and try to cherry > > pick > > > > > all of those before tagging the next release c
Re: New Pulsar website goes live!
Hi Yu, Thank you and all the other contributors for all the great work! Well done. - Sijie On Wed, Feb 16, 2022 at 8:04 PM Yu wrote: > Thanks for your quick feedback, Penghui and Dianjin. > > As mentioned in the last email, "*...**there are some known issues (eg. > localized pages and some CLI pages are not available). We're fixing them* > *...* " > > We'll keep you updated once we resolve them. > > On Thu, Feb 17, 2022 at 11:52 AM Dianjin Wang > wrote: > > > Great work! > > > > Any plans for multi-language translation support? There were some > languages > > supported, such as Chinese, Japanese, French and more, but now cannot > find > > the button in the new website. > > > > Best, > > Dianjin Wang > > > > > > On Thu, Feb 17, 2022 at 11:15 AM Yu wrote: > > > > > Hi Pulsarers, > > > > > > As you may notice, the new Pulsar website goes live! > > > https://pulsar.apache.org/ > > > > > > We hope you enjoy the new feature and the fresh appearance! > > > > > > CHANGE: documentation main page URL is changed from > > > https://pulsar.apache.org/docs/en/next/standalone/ to > > > https://pulsar.apache.org/docs/next/. > > > > > > It does not affect user experience if you read documentation navigated > > > from the Pulsar main page. > > > > > > Besides, there are some known issues (eg. localized pages and some CLI > > > pages are not available). We're fixing them. Feel free to join us or > make > > > comments, thanks! > > > > > > Special thanks go to @urfreespace and @dave2wave to make this happen! > > > > > > > > > Anonymitaet > > > > > >
Re: [VOTE] Pulsar Release 2.8.3 Candidate 1
Thanks, Michael, looking forward to future collaboration. Best, Penghui On Thu, Feb 17, 2022 at 12:37 PM Michael Marshall wrote: > Thanks for your reply, Penghui. I saw your note on that thread but > didn't realize it affected 2.8.3. > > I absolutely did not mean to imply that you don't want to share here. > I appreciate your many contributions to the community and the mailing > list, and I look forward to future collaboration. > > Thanks, > Michael > > > On Wed, Feb 16, 2022 at 6:58 PM PengHui Li wrote: > > > > > I agree that it can be non-trivial to determine if an issue is an > > internal test or an upstream issue. > > > > My main point is that we should prefer early communication of > > potential issues that affect pending releases and accept that the > > issue might prove to be nothing more than a short delay in the release > > cycle. The alternative is that a release manager completes the build > > process, which takes several hours, then discovers a new candidate is > > needed. There is still an inherent race condition for release creation > > and validation, but early communication (and collaboration) might help > > us avoid some of the cases where candidates are rejected immediately. > > > > I have explained here > > https://lists.apache.org/thread/m5so1scfok44hnd74436w100smn2vbh3 > > at 11/02/2022 > > Have you seen it? As I said before, I forget to update this thread, > > I have declared at the first time that I respond to this thread. > > it's not that I don't want to share to this thread. > > I can't accept dave and you thinking I don't want to share here. > > it's completely different > > > > > For what it's worth, I don't think you need to verify your test > > correctness 100% before sharing potential issues that affect pending > > releases. A quick note to the dev mailing list indicating that you're > > investigating a potential issue would be sufficient and valuable. > > > > If I can't confirm the issue, what should I share? > > Our failed tests happen every day due to different problems, > > do I need to share them with the community every day? > > > > Penghui > > > > On Thu, Feb 17, 2022 at 3:41 AM Michael Marshall > > wrote: > > > > > I agree that it can be non-trivial to determine if an issue is an > > > internal test or an upstream issue. > > > > > > My main point is that we should prefer early communication of > > > potential issues that affect pending releases and accept that the > > > issue might prove to be nothing more than a short delay in the release > > > cycle. The alternative is that a release manager completes the build > > > process, which takes several hours, then discovers a new candidate is > > > needed. There is still an inherent race condition for release creation > > > and validation, but early communication (and collaboration) might help > > > us avoid some of the cases where candidates are rejected immediately. > > > > > > For what it's worth, I don't think you need to verify your test > > > correctness 100% before sharing potential issues that affect pending > > > releases. A quick note to the dev mailing list indicating that you're > > > investigating a potential issue would be sufficient and valuable. > > > > > > Thanks, > > > Michael > > > > > > > > > > > > > > > > > > On Tue, Feb 15, 2022 at 6:30 PM PengHui Li wrote: > > > > > > > > > Just a friendly reminder that we are an open source project and > other > > > > community members who may have different internal testing may find > better > > > > fixes. > > > > > > > > Sorry, dave. I know we are an open source project. > > > > We found a problem in our internal test, we need to confirm it is not > > > > the test issue right? > > > > > > > > Do you suggest that we discuss this in the email when we find our > > > internal > > > > test errors? > > > > The community has no responsibility to deal with our internal testing > > > bugs. > > > > > > > > On Wed, Feb 16, 2022 at 8:04 AM Dave Fisher wrote: > > > > > > > > > > > > > > > > > > > > On Feb 15, 2022, at 3:59 PM, PengHui Li > wrote: > > > > > > > > > > > >> I wonder if we should be more willing to share suspicions on the > > > > > > mailing list when releases are actively being prepared. I would > > > > > > have appreciated the notice, and I could have possibly helped > > > > > > confirm the issue. On the other hand, I respect that it is hard > to > > > > > > know when to share a theory and when to test it out a bit more. > > > > > > > > > > > > Yes, It's just that these issues are happening in our internal > > > testing > > > > > > and we need to 100% confirm the issue and then share it out, > > > > > > otherwise, We may confuse the contributors due to internal test > > > problems. > > > > > > > > > > Just a friendly reminder that we are an open source project and > other > > > > > community members who may have different internal testing may find > > > better > > > > > fixes. > > > > > > > > > > > > > > > > > This kind of problem is often easy to fix,
Pulsar Helm Chart Repo is broken
Hi all, I think the new website push has broken the Pulsar Helm Chart repo. We might need to take a look at how to fix it. - Sijie
Re: [VOTE] Pulsar Release 2.8.3 Candidate 2
Thanks for pointing that out, Penghui. I can confirm that my KEY is not signed. While it's not the same as in person verification, I'll note that since becoming a committer I have signed all of my git commits with the same gpg key used to sign these release artifacts. At the very least, we can be confident that the author of those commits is also the author of these artifacts. :) Thanks, Michael On Wed, Feb 16, 2022 at 9:05 PM PengHui Li wrote: > > Thanks for the explanation Dave. > > Best, > Penghui > > On Thu, Feb 17, 2022 at 10:37 AM Dave Fisher wrote: > > > While it’s preferred that KEYS are signed it’s not required. The last > > Apache key signing party was at Apachecon Las Vegas in 2019. Here’s looking > > forward to the next one. Bring your government ids. > > > > Regards, > > Dave > > > > Sent from my iPhone > > > > > On Feb 16, 2022, at 6:11 PM, PengHui Li wrote: > > > > > > Hi Michael, > > > > > > Please check your key > > > > > > ``` > > > gpg: assuming signed data in 'apache-pulsar-2.8.3-bin.tar.gz' > > > gpg: Signature made Wed Feb 16 04:49:42 2022 CST > > > gpg:using RSA key > > BD4291E509D771B79E7BD1F5C5724B3F5588C4EB > > > gpg: Good signature from "Michael Marshall " > > [unknown] > > > gpg: WARNING: This key is not certified with a trusted signature! > > > gpg: There is no indication that the signature belongs to the > > > owner. > > > ``` > > > > > > I think it should be an issue that the key does not been signed? I'm not > > > sure. > > > > > > Thanks, > > > Penghui > > > > > >> On Thu, Feb 17, 2022 at 9:56 AM PengHui Li wrote: > > >> > > >> +1 (binding) > > >> > > >> - Checked the signature > > >> - Start the standalone > > >> - Publish and consume messages > > >> - Check the Cassandra connector > > >> - Check the stateful function > > >> > > >> Thanks for the great work, > > >> Penghui > > >> > > >> On Wed, Feb 16, 2022 at 12:35 PM Michael Marshall > > > > >> wrote: > > >> > > >>> This is the second release candidate for Apache Pulsar, version 2.8.3. > > >>> > > >>> It fixes the following issues: > > >>> https://github.com/apache/pulsar/compare/v2.8.2...v2.8.3-candidate-2 > > >>> > > >>> *** Please download, test and vote on this release. This vote will stay > > >>> open > > >>> for at least 72 hours *** > > >>> > > >>> Note that we are voting upon the source (tag), binaries are provided > > for > > >>> convenience. > > >>> > > >>> Source and binary files: > > >>> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.3-candidate-2/ > > >>> > > >>> SHA-512 checksums: > > >>> > > >>> > > >>> > > aa9fb934260e158fe6a30208324459dd747d32fbb52ee61a28a322c6161f3e21d2e97f61118ee0e82488720f7c7787233949ef9eb80567b83896f08b12c54090 > > >>> ./apache-pulsar-2.8.3-bin.tar.gz > > >>> > > >>> > > 47747e2cdc323c00fcdd08c537c77f355a94fbfaee77789718cb70e52726b7084522a842505e7eff1c1be26fddc850f2134d400a803854286ac8ea2f7cada121 > > >>> ./apache-pulsar-2.8.3-src.tar.gz > > >>> > > >>> Unofficial Docker images: > > >>> michaelmarshall/pulsar:2.8.3-rc2 > > >>> michaelmarshall/pulsar-all:2.8.3-rc2 > > >>> michaelmarshall/pulsar-standalone:2.8.3-rc2 > > >>> michaelmarshall/pulsar-grafana:2.8.3-rc2 > > >>> > > >>> Maven staging repo: > > >>> > > https://repository.apache.org/content/repositories/orgapachepulsar-1140/ > > >>> > > >>> The tag to be voted upon: > > >>> v2.8.3-candidate-2 (83a522f3a17d41eb3727ffee67cdf035e8ea471b) > > >>> https://github.com/apache/pulsar/releases/tag/v2.8.3-candidate-2 > > >>> > > >>> Pulsar's KEYS file containing PGP keys we use to sign the release: > > >>> https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > >>> > > >>> Please download the source package, and follow the README to build > > >>> and run the Pulsar standalone service. > > >>> > > >> > > > >
Re: Pulsar Helm Chart Repo is broken
I send a notice to users@ mailing list and #general channel about the situation. - Sijie On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo wrote: > Hi all, > > I think the new website push has broken the Pulsar Helm Chart repo. > > We might need to take a look at how to fix it. > > - Sijie >
[NOTICE] Pulsar Helm Charts Repo is temporarily down
Hi everyone, Just want to let everyone know that the Pulsar Helm Charts repo is temporarily down due to the new Pulsar website push. The `charts/index.yaml` was not added to the new Pulsar site. So if you are using the helm command to pull the charts, it will fail. A bunch of the committers is looking into the issue now. We will keep everyone posted on the progress. - Sijie
Re: Pulsar Helm Chart Repo is broken
That seems to be because the new site switch today is causing this exception, we should update the helm chart ci to fix this issue, will take a look at it later Thanks, Guangning Sijie Guo 于2022年2月17日周四 12:52写道： > I send a notice to users@ mailing list and #general channel about the > situation. > > - Sijie > > On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo wrote: > > > Hi all, > > > > I think the new website push has broken the Pulsar Helm Chart repo. > > > > We might need to take a look at how to fix it. > > > > - Sijie > > >
Re: [NOTICE] Pulsar Helm Charts Repo is temporarily down
We can revert the PRs that switched over if necessary and follow with more testing. It’s a trade off. All the best, Dave Sent from my iPhone > On Feb 16, 2022, at 8:54 PM, Sijie Guo wrote: > > Hi everyone, > > Just want to let everyone know that the Pulsar Helm Charts repo is > temporarily down due to the new Pulsar website push. The > `charts/index.yaml` was not added to the new Pulsar site. So if you are > using the helm command to pull the charts, it will fail. A bunch of the > committers is looking into the issue now. We will keep everyone posted on > the progress. > > - Sijie
Re: Pulsar Helm Chart Repo is broken
Yes, we just need to change the CI process in `pulsar-helm-chart` to publish the index.yaml file to the new website. See https://github.com/apache/pulsar-helm-chart/blob/master/.ci/release.sh#L72 On Wed, Feb 16, 2022 at 8:55 PM Guangning E wrote: > That seems to be because the new site switch today is causing this > exception, we should update the helm chart ci to fix this issue, will take > a look at it later > > Thanks, > Guangning > > Sijie Guo 于2022年2月17日周四 12:52写道： > > > I send a notice to users@ mailing list and #general channel about the > > situation. > > > > - Sijie > > > > On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo wrote: > > > > > Hi all, > > > > > > I think the new website push has broken the Pulsar Helm Chart repo. > > > > > > We might need to take a look at how to fix it. > > > > > > - Sijie > > > > > >
[GitHub] [pulsar-helm-chart] tuteng opened a new pull request #231: Fixed release yaml file
tuteng opened a new pull request #231: URL: https://github.com/apache/pulsar-helm-chart/pull/231 Fixes # ### Motivation This update fixes an issue with the site release that caused the helm chart to no longer be used. Exception: ``` ...Unable to get an update from the "apache-pulsar" chart repository (https://pulsar.apache.org/charts): failed to fetch https://pulsar.apache.org/charts/index.yaml : 404 Not Found ``` ### Modifications * Update release yaml file ### Verifying this change - [ ] Make sure that the change passes the CI checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: Pulsar Helm Chart Repo is broken
Change L74 to pulsar-site Change L76 to asf-site-next All the best, Dave Sent from my iPhone > On Feb 16, 2022, at 9:02 PM, Sijie Guo wrote: > > Yes, we just need to change the CI process in `pulsar-helm-chart` to > publish the index.yaml file to the new website. > > See > https://github.com/apache/pulsar-helm-chart/blob/master/.ci/release.sh#L72 > >> On Wed, Feb 16, 2022 at 8:55 PM Guangning E wrote: >> >> That seems to be because the new site switch today is causing this >> exception, we should update the helm chart ci to fix this issue, will take >> a look at it later >> >> Thanks, >> Guangning >> >> Sijie Guo 于2022年2月17日周四 12:52写道： >> >>> I send a notice to users@ mailing list and #general channel about the >>> situation. >>> >>> - Sijie >>> On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo wrote: >>> Hi all, I think the new website push has broken the Pulsar Helm Chart repo. We might need to take a look at how to fix it. - Sijie >>> >>
Re: Pulsar Helm Chart Repo is broken
yes, this pr for fixed it => https://github.com/apache/pulsar-helm-chart/pull/231 Dave Fisher 于2022年2月17日周四 13:21写道： > Change L74 to pulsar-site > > Change L76 to asf-site-next > > All the best, > Dave > > Sent from my iPhone > > > On Feb 16, 2022, at 9:02 PM, Sijie Guo wrote: > > > > Yes, we just need to change the CI process in `pulsar-helm-chart` to > > publish the index.yaml file to the new website. > > > > See > > > https://github.com/apache/pulsar-helm-chart/blob/master/.ci/release.sh#L72 > > > >> On Wed, Feb 16, 2022 at 8:55 PM Guangning E > wrote: > >> > >> That seems to be because the new site switch today is causing this > >> exception, we should update the helm chart ci to fix this issue, will > take > >> a look at it later > >> > >> Thanks, > >> Guangning > >> > >> Sijie Guo 于2022年2月17日周四 12:52写道： > >> > >>> I send a notice to users@ mailing list and #general channel about the > >>> situation. > >>> > >>> - Sijie > >>> > On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo wrote: > >>> > Hi all, > > I think the new website push has broken the Pulsar Helm Chart repo. > > We might need to take a look at how to fix it. > > - Sijie > > >>> > >> >
[GitHub] [pulsar-helm-chart] lhotari opened a new pull request #232: [Build] Publish charts to apache/pulsar-site branch asf-site-next
lhotari opened a new pull request #232: URL: https://github.com/apache/pulsar-helm-chart/pull/232 - also use shallow cloning -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] urfreespace commented on a change in pull request #231: Fixed release yaml file
urfreespace commented on a change in pull request #231:
URL: https://github.com/apache/pulsar-helm-chart/pull/231#discussion_r808689895
##
File path: .ci/release.sh
##
@@ -71,9 +71,9 @@ function release::git_setup() {
function release::publish_charts() {
release::git_setup
-git clone https://${GITHUB_TOKEN}@github.com/apache/pulsar
-cd pulsar
-git checkout asf-site
+git clone https://${GITHUB_TOKEN}@github.com/apache/pulsar-site
Review comment:
the default branch is `asf-site` for now, However, the JIRA has been
submitted for modification, it's recommend change it to `git clone -b main
https://${GITHUB_TOKEN}@github.com/apache/pulsar-site` @tuteng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-helm-chart] urfreespace commented on a change in pull request #231: Fixed release yaml file
urfreespace commented on a change in pull request #231:
URL: https://github.com/apache/pulsar-helm-chart/pull/231#discussion_r808690607
##
File path: .ci/release.sh
##
@@ -71,9 +71,9 @@ function release::git_setup() {
function release::publish_charts() {
release::git_setup
-git clone https://${GITHUB_TOKEN}@github.com/apache/pulsar
-cd pulsar
-git checkout asf-site
+git clone https://${GITHUB_TOKEN}@github.com/apache/pulsar-site
Review comment:
Of course, it doesn't matter, because you switched branches to
`asf-site-next`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-helm-chart] tuteng commented on pull request #231: Fixed release yaml file
tuteng commented on pull request #231: URL: https://github.com/apache/pulsar-helm-chart/pull/231#issuecomment-1042593947 It is duplicated with pr #232 and I will close it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] tuteng closed pull request #231: Fixed release yaml file
tuteng closed pull request #231: URL: https://github.com/apache/pulsar-helm-chart/pull/231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari merged pull request #232: [Build] Publish charts to apache/pulsar-site branch asf-site-next
lhotari merged pull request #232: URL: https://github.com/apache/pulsar-helm-chart/pull/232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari merged pull request #223: Improve Zookeeper "ruok" probes: use TLS port when TLS is enabled, specify "-q 1" for nc
lhotari merged pull request #223: URL: https://github.com/apache/pulsar-helm-chart/pull/223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on a change in pull request #223: Improve Zookeeper "ruok" probes: use TLS port when TLS is enabled, specify "-q 1" for nc
lhotari commented on a change in pull request #223:
URL: https://github.com/apache/pulsar-helm-chart/pull/223#discussion_r808695020
##
File path: charts/pulsar/templates/zookeeper-statefulset.yaml
##
@@ -151,7 +157,9 @@ spec:
command:
- timeout
- "{{ .Values.zookeeper.probe.readiness.timeoutSeconds }}"
-- bin/pulsar-zookeeper-ruok.sh
+- bash
Review comment:
Thanks. I added this to the commit message
1c4f745941fda76e8ec532bc734b8499fe442b25 since it's easier to find the
information there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: Pulsar Helm Chart Repo is broken
The chart index is now published, https://pulsar.apache.org/charts/index.yaml . However, the history isn't there. I'll fix that with a manual commit. -Lari On Thu, Feb 17, 2022 at 7:31 AM Guangning E wrote: > yes, this pr for fixed it => > https://github.com/apache/pulsar-helm-chart/pull/231 > > Dave Fisher 于2022年2月17日周四 13:21写道： > > > Change L74 to pulsar-site > > > > Change L76 to asf-site-next > > > > All the best, > > Dave > > > > Sent from my iPhone > > > > > On Feb 16, 2022, at 9:02 PM, Sijie Guo wrote: > > > > > > Yes, we just need to change the CI process in `pulsar-helm-chart` to > > > publish the index.yaml file to the new website. > > > > > > See > > > > > > https://github.com/apache/pulsar-helm-chart/blob/master/.ci/release.sh#L72 > > > > > >> On Wed, Feb 16, 2022 at 8:55 PM Guangning E > > wrote: > > >> > > >> That seems to be because the new site switch today is causing this > > >> exception, we should update the helm chart ci to fix this issue, will > > take > > >> a look at it later > > >> > > >> Thanks, > > >> Guangning > > >> > > >> Sijie Guo 于2022年2月17日周四 12:52写道： > > >> > > >>> I send a notice to users@ mailing list and #general channel about > the > > >>> situation. > > >>> > > >>> - Sijie > > >>> > > On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo > wrote: > > >>> > > Hi all, > > > > I think the new website push has broken the Pulsar Helm Chart repo. > > > > We might need to take a look at how to fix it. > > > > - Sijie > > > > >>> > > >> > > >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
Thanks for creating the branch, Penghui. > Yes, but I think that the code freeze is only meaningful if it’s > communicated in advance. Given that our community members who focus on testing are otherwise about to prepare for a quick 3 day round of testing, I don't believe they would object to a last minute change that gives them more time for testing. In your view, who needs this advanced communication to make our code freeze meaningful? Thanks, Michael On Wed, Feb 16, 2022 at 8:07 PM PengHui Li wrote: > > Hi all, > > Put an update here, I have created branch-2.10[1] > > [1]https://github.com/apache/pulsar/tree/branch-2.10 > > On Thu, Feb 17, 2022 at 9:23 AM PengHui Li wrote: > > > Hi Micheal > > > > > Penghui, is your current plan to create branch-2.10, create the > > release artifacts, and start a vote on them all within a few days of > > each other? > > > > Yes, I will create branch-2.10 today. > > > > For starting the vote, we need to confirm these 2 PRs[1] will not introduce > > breaking changes. Very grateful if someone can also help verify them. > > > > > > [1] https://github.com/apache/pulsar/pull/13376, > > https://github.com/apache/pulsar/pull/13341 > > > > Thanks, > > Penghui > > > > On Thu, Feb 17, 2022 at 8:59 AM Matteo Merli > > wrote: > > > >> Yes, but I think that the code freeze is only meaningful if it’s > >> communicated in advance. > >> > >> The fact that it was included in the original PIP but never followed in > >> the > >> practice means it would be a last minute change. > >> > >> On Wed, Feb 16, 2022 at 2:37 PM Michael Marshall > >> wrote: > >> > >> > When we discussed the code freeze in the community meeting on 2/3, I > >> > was under the impression that it was a new development to our existing > >> > release process. I subsequently learned it was already defined in > >> > PIP 47. Even if we haven't been following this part of PIP 47, what > >> > is the value in waiting until 2.11 to follow our already defined > >> process? > >> > While I agree it is helpful to provide guidance on when a version will > >> > ship, > >> > I think it is more important to give the community time to test a > >> release, > >> > even if that means we're a little late on our release schedule. So far, > >> > we haven't even created a branch to begin testing. > >> > > >> > Note also that Sijie suggested using a feature freeze early on in this > >> > thread. > >> > > >> > The 2.9.0 release is relevant here. It had 4 release candidates over 4 > >> > weeks and the final result was broken. That indicates to me that tagging > >> > an RC early does not guarantee an early release and that our current > >> > process isn't optimal and likely needs adjustments. I do not think we > >> > should wait to address these issues. I propose we start following > >> > PIP 47's guidance on code freeze and release stabilization periods. > >> > > >> > > I don't think that changes the picture here. There are *always* last > >> > > minute issues being discovered, and there is a call to be made on a > >> > > case by case. The feature freeze will reduce the likelihood of > >> > > introducing *more* issues by getting it from the master branch, but > >> > > won't change a comma from issues that were already there. > >> > > >> > I thought you wanted to implement a code/feature freeze to allow for > >> > more release stabilization. Can you clarify what you mean here? > >> > > >> > Thanks, > >> > Michael > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Wed, Feb 16, 2022 at 2:42 PM Matteo Merli > >> > wrote: > >> > > > >> > > Michael, as we chatted in last weekly meeting (though not yet > >> > > formalized), since we have never really done a feature freeze on the > >> > > branch during paste releases, we should start from the next release, > >> > > to give a decent preview of what to expect to developers in terms of > >> > > dates. > >> > > > >> > > > While some may feel "behind" in getting out the 2.10 release, our > >> > > > priority must be to give the community time to verify the stability > >> of > >> > > > the release. > >> > > > >> > > I don't think that changes the picture here. There are *always* last > >> > > minute issues being discovered, and there is a call to be made on a > >> > > case by case. The feature freeze will reduce the likelihood of > >> > > introducing *more* issues by getting it from the master branch, but > >> > > won't change a comma from issues that were already there. > >> > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Matteo Merli > >> > > > >> > > > >> > > On Wed, Feb 16, 2022 at 10:47 AM Michael Marshall < > >> mmarsh...@apache.org> > >> > wrote: > >> > > > > >> > > > > I will build the release and start the vote before next > >> Monday(GMT+8) > >> > > > > >> > > > Penghui, is your current plan to create branch-2.10, create the > >> > > > release artifacts, and start a vote on them all within a few days of > >> > > > each other? > >> > > > > >> > > > > I'm doing my best to follow PIP 4
Re: Pulsar Helm Chart Repo is broken
Chart history added to https://pulsar.apache.org/charts/index.yaml With commit https://github.com/apache/pulsar-site/commit/f45d4f89c13fe25802f374f8d5c015334690c37f Please append something to the url if you get the old version when previewing the chart index in the browser, for example https://pulsar.apache.org/charts/index.yaml?dummy . GitHub pages caching is explained in this SO answer: https://stackoverflow.com/questions/24851824/how-long-does-it-take-for-github-page-to-show-changes-after-changing-index-html . -Lari On Thu, Feb 17, 2022 at 7:52 AM Lari Hotari wrote: > The chart index is now published, > https://pulsar.apache.org/charts/index.yaml . > However, the history isn't there. I'll fix that with a manual commit. > > -Lari > > On Thu, Feb 17, 2022 at 7:31 AM Guangning E wrote: > >> yes, this pr for fixed it => >> https://github.com/apache/pulsar-helm-chart/pull/231 >> >> Dave Fisher 于2022年2月17日周四 13:21写道： >> >> > Change L74 to pulsar-site >> > >> > Change L76 to asf-site-next >> > >> > All the best, >> > Dave >> > >> > Sent from my iPhone >> > >> > > On Feb 16, 2022, at 9:02 PM, Sijie Guo wrote: >> > > >> > > Yes, we just need to change the CI process in `pulsar-helm-chart` to >> > > publish the index.yaml file to the new website. >> > > >> > > See >> > > >> > >> https://github.com/apache/pulsar-helm-chart/blob/master/.ci/release.sh#L72 >> > > >> > >> On Wed, Feb 16, 2022 at 8:55 PM Guangning E >> > wrote: >> > >> >> > >> That seems to be because the new site switch today is causing this >> > >> exception, we should update the helm chart ci to fix this issue, will >> > take >> > >> a look at it later >> > >> >> > >> Thanks, >> > >> Guangning >> > >> >> > >> Sijie Guo 于2022年2月17日周四 12:52写道： >> > >> >> > >>> I send a notice to users@ mailing list and #general channel about >> the >> > >>> situation. >> > >>> >> > >>> - Sijie >> > >>> >> > On Wed, Feb 16, 2022 at 8:46 PM Sijie Guo >> wrote: >> > >>> >> > Hi all, >> > >> > I think the new website push has broken the Pulsar Helm Chart repo. >> > >> > We might need to take a look at how to fix it. >> > >> > - Sijie >> > >> > >>> >> > >> >> > >> >
Re: New Feature Proposal: flag to require a non-partitioned topic on Producer/Consumer Builder
> Ideally, people shouldn't be aware of partitioned vs non-partitioned. I'm not sure I agree that people shouldn't be aware of these details considering the two types of topics provide different guarantees regarding ordering. Enrico's request is essentially to have a way to fail fast when a topic is partitioned. Note that the logic for whether or not a topic is partitioned is already available to the client: that is how it knows whether to create a partitioned producer or a non-partitioned producer. The simplest solution is to add a method to the client named "createNonPartitionedProducer". The method would then fail if the topic lookup results in a topic with more than 0 partitions. Alternatively, you could create a custom `MessageRouter` that always picks the 0 partition when `choosePartition` is called. However, I view this as a bit of a hack and think my first solution is a better fit for Enrico's request. Thanks, Michael On Wed, Feb 16, 2022 at 3:27 PM Sijie Guo wrote: > > I don't think we should expose this feature flag on the client-side. > Ideally, people shouldn't be aware of partitioned vs non-partitioned. > > - Sijie > > On Wed, Feb 16, 2022 at 1:37 AM Enrico Olivelli wrote: > > > Ping > > > > Il Dom 13 Feb 2022, 11:49 Enrico Olivelli ha > > scritto: > > > > > Hello, > > > I have a use case in which my application MUST use a non-partitioned > > > topic to work properly, but the topic name is (of course) configurable > > > by the user. > > > If I am not using a non-partitioned topic, I do not have string > > > guarantees on the ordering of the message (because messages will be > > > spread across multiple partitions). > > > > > > Currently there is no way to require that the topic IS a > > > NON-PARTITIONED topic. Especially when the topic does not exist yet, > > > you configured the namespace to create partitioned topics by default. > > > > > > Using a PulsarAdmin preliminary call is not a good workaround because > > > it is expensive, and ideally I want to be able to create the Producer > > > (or the Consumer) and see that everything works automatically, failing > > > in case of a partitioned topic. > > > > > > Thoughts ? > > > > > > > > > Enrico > > > > >
Re: [DISCUSS] Apache Pulsar 2.10.0 release
On Wed, Feb 16, 2022 at 9:55 PM Michael Marshall wrote: > Given that our community members who focus on testing are otherwise > about to prepare for a quick 3 day round of testing, I don't believe > they would object to a last minute change that gives them more time > for testing. In your view, who needs this advanced communication to > make our code freeze meaningful? That developers know when is the cutoff date and can plan accordingly.
RE: [VOTE] Pulsar Client Go Release 0.8.0 Candidate 1
+1 (non-binding) - checked checksum and signature - run produce and consume Thanks, Shusuke Tsuda -Original Message- From: Chris Kellogg Sent: Thursday, February 17, 2022 12:53 PM To: dev@pulsar.apache.org Subject: Re: [VOTE] Pulsar Client Go Release 0.8.0 Candidate 1 +1 Checked the signatures, complied and ran the pular perf tool (produce and consume) against a standalone cluster. On Wed, Feb 16, 2022 at 5:29 PM PengHui Li wrote: > > +1 > > - Checked the signatures > - Run produce and consume on a standalone > > Penghui > > On Thu, Feb 17, 2022 at 12:45 AM Matteo Merli > wrote: > > > +1 > > > > Checked signatures, compiled and run perf tool. > > > > > > > > -- > > Matteo Merli > > > > > > On Tue, Feb 15, 2022 at 7:28 PM r...@apache.org > > > > wrote: > > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version > > > 0.8.0, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific > > > comments) > > > > > > This is the first release candidate for Apache Pulsar Go client, > > > version > > 0.8.0. > > > > > > It fixes the following issues: > > > https://github.com/apache/pulsar-client-go/milestone/9?closed=1 > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign > > > this > > release: > > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > dist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7 > > > C01%7Cshustsud%40yahoo-corp.jp%7C3defcafff6914e6349e108d9f1c9176d% > > > 7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637806669593447909%7C > > > Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6 > > > Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Xxgx5Ja22qYNA%2FJF9T5tteCQ > > > %2FcLNmQEVEXi%2Bl%2BaLxls%3D&reserved=0 > > > > > > Please download these packages and review this release candidate: > > > - Review release notes > > > - Download the source package (verify shasum, and asc) and follow > > > the README.md to build and run the pulsar-client-go. > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > majority approval, with at least 3 PMC affirmative votes. > > > > > > Source file: > > > > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdi > > st.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-client-go-0.8.0 > > -candidate-1%2F&data=04%7C01%7Cshustsud%40yahoo-corp.jp%7C3defca > > fff6914e6349e108d9f1c9176d%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C > > 0%7C637806669593447909%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL > > CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lz%2B > > iD13jAtcOLLHdP0S7KTGIhz7t3Hj1aMAzhEifZg0%3D&reserved=0 > > > > > > The tag to be voted upon: > > > v0.8.0 > > > https://github.com/apache/pulsar-client-go/tree/v0.8.0-candidate-1 > > > > > > SHA-512 checksums: > > > > > 0015cc735af13139a564f0fd57c58af753579953e3c12f08cb62d8e9801cdc2964c6 > > 0392ed9a98dd54845be7f7b1a353bd49cecb03360b4cb691904f8447769c > > > apache-pulsar-client-go-0.8.0-src.tar.gz > >
Re: [ANNOUNCE] New Committer: Li Li
Congratulations Haiting Jiang 于2022年2月16日周三 22:12写道： > Congrats! > > Thanks, > Haiting > > On 2022/02/16 10:51:52 Guangning E wrote: > > Configurations! > > > > > > Thanks, > > Guangning > > > > Jia Zhai 于2022年2月16日周三 15:40写道： > > > > > Congratulations > > > > > > > > > On Wed, Feb 16, 2022 at 11:39 AM Yu wrote: > > > > > > > Hi Lili, > > > > > > > > Congrats! Well deserved! You've made a lot of great changes to the > Pulsar > > > > website! > > > > > > > > On Tue, Feb 15, 2022 at 5:56 PM ZhangJian He > wrote: > > > > > > > > > Congratulations > > > > > > > > > > Thanks > > > > > ZhangJian He > > > > > > > > > > Huanli Meng 于2022年2月15日周二 > 17:03写道： > > > > > > > > > > > Congratulations Li. > > > > > > > > > > > > BR// Huanli > > > > > > > > > > > > > On Feb 15, 2022, at 3:06 PM, PengHui Li > > > wrote: > > > > > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has > invited Li > > > > Li > > > > > > > https://github.com/urfreespace to become a committer and we > are > > > > > pleased > > > > > > to > > > > > > > announce that he has accepted. > > > > > > > > > > > > > > Li Li has done a great contribution to Pulsar Website, > > > documentation. > > > > > > > > > > > > > > Welcome and Congratulations, Li Li! > > > > > > > > > > > > > > Please join us in congratulating and welcoming Li Li onboard! > > > > > > > > > > > > > > Best Regards, > > > > > > > Penghui Li on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] PIP-118: Do not restart brokers when ZooKeeper session expires
I forgot to close this one. Closing now with 5 +1s votes; * Matteo * PengHui * Hang * Enrico * Sijie -- Matteo Merli On Mon, Jan 24, 2022 at 11:47 PM Enrico Olivelli wrote: > > What's the status of this PIP ? > > We have a pending PR, > we can commit it as soon as the VOTE thread is officially closed > https://github.com/apache/pulsar/pull/13341 > > > Il giorno gio 23 dic 2021 alle ore 01:17 Sijie Guo > ha scritto: > > > > +1 > > > > On Tue, Dec 21, 2021 at 3:22 PM Matteo Merli wrote: > > > > > https://github.com/apache/pulsar/issues/13304 > > > > > > Following the discussion, I have updated the proposal to also include > > > the deprecation and renaming of the config setting name to > > > `metadataSessionExpiredPolicy`. > > > > > > > > > > > > --- > > > ## Motivation > > > > > > After all the work done for PIP-45 that was already included in 2.8 and > > > 2.9 > > > releases, it enabled the concept of re-acquirable resource locks and > > > leader > > > election. > > > > > > Another important change was to avoid doing any deferrable metadata > > > operation > > > when we know that we are not currently connected to the metadata service. > > > > > > Finally, that enabled to stabilize in 2.9 the configuration setting that > > > allows > > > brokers to continue operating in a safe mode when the session with > > > ZooKeeper > > > expires. > > > > > > The way it works is that, when we lose a ZooKeeper session, the data plane > > > will > > > continue to work undisturbed, relying on the BookKeeper fencing to avoid > > > any > > > inconsistencies. > > > > > > New topics are not able to get started, but existing topics will see no > > > impact. > > > > > > The original intention for shutting down the brokers was to ensure that we > > > would automatically go back to a consistent state, with respect to which > > > resources are "owned" in ZooKeeper by a given broker. > > > > > > With the re-acquirable resource locks, that problem was solved and > > > thoroughly > > > tested to be robust. > > > > > > ## Proposed changes > > > > > > In 2.10 release, for the setting: > > > > > > ```properties > > > # There are two policies to apply when broker metadata session > > > expires: session expired happens, "shutdown" or "reconnect". > > > # With "shutdown", the broker will be restarted. > > > # With "reconnect", the broker will keep serving the topics, while > > > attempting to recreate a new session. > > > zookeeperSessionExpiredPolicy=shutdown > > > ``` > > > > > > Deprecate the old setting and rename it to > > > `metadataSessionExpiredPolicy`, with default value set to `reconnect`. > > >
Re: [VOTE] Pulsar Release 2.9.2 Candidate 3
+1 (binding) - Check signatures/checksums - Check the license headers - build sources - Validate Pub/Sub and Java Functions - Validate Stateful Functions Regards, Hiroyuki From: Ran Gao Sent: Thursday, February 17, 2022 03:26 To: dev@pulsar.apache.org Subject: [VOTE] Pulsar Release 2.9.2 Candidate 3 This is the third release candidate for Apache Pulsar, version 2.9.2. *** Please download, test, and vote on this release. This vote will stay open for at least 72 hours *** Note that we are voting upon the source (tag), binaries are provided for convenience. Source and binary files: https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-2.9.2-candidate-3%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C2ffeea75984e4f45525008d9f179eba0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637806328206378803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nkVBN0W%2FMQWvcQl6qkCfVxviEYd42j1oZ5xL8uBQcSo%3D&reserved=0 SHA-512 checksums: 21ab21df057dc181d02806e67a8d85b861fa60a8413a8e4058ffcb738d9ef81d8797c4e7249d50cf2a23c7437cb2880f4240d010a983eda33a2071f5fddf6b01 apache-pulsar-2.9.2-bin.tar.gz f927d77aba32d1112cd009a1d91a8fca27a49fcb20ea215a5792dc16e2b04bd822238c3f29dd81a7853e20062a0454e2abe608626b24e4e25ff06915b7a04785 apache-pulsar-2.9.2-src.tar.gz Maven staging repo: https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachepulsar-1141%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C2ffeea75984e4f45525008d9f179eba0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637806328206378803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WtUAnQi5BA8SE3T%2FhiOvO8AelynLS6kFOkX0hWwbLhg%3D&reserved=0 The tag to be voted upon: v2.9.2-candidate-3 (5615809ffd68599ae7fec1be0679ee83171f42dc) https://github.com/apache/pulsar/releases/tag/v2.9.2-candidate-3 Pulsar's KEYS file containing PGP keys we use to sign the release: https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C2ffeea75984e4f45525008d9f179eba0%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637806328206378803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Uu92f4VV8sGmffdA0YoUXb39J1AYtfRk%2FoFPNLbGeE4%3D&reserved=0 Please download the source package, and follow the README to build and run the Pulsar standalone service.
[GitHub] [pulsar-helm-chart] lhotari commented on pull request #190: Bump to Pulsar 2.8.2
lhotari commented on pull request #190: URL: https://github.com/apache/pulsar-helm-chart/pull/190#issuecomment-1042658696 Rebased on latest master branch changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari commented on issue #164: 2.7.3 helm chart still use 2.7.2 images
lhotari commented on issue #164: URL: https://github.com/apache/pulsar-helm-chart/issues/164#issuecomment-1042660744 The helm chart version is independent from Pulsar's version. There's a separate PR #200 that would make image tag match the appVersion by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-helm-chart] lhotari closed issue #164: 2.7.3 helm chart still use 2.7.2 images
lhotari closed issue #164: URL: https://github.com/apache/pulsar-helm-chart/issues/164 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
