Re: [VOTE] Pulsar Release 2.8.4 Candidate 1
Thank you all, Close the vote with 3 (+1) bindings and 2 (+1) non-bindings. I am closing this vote and will continue the release process. Thanks, Yunze > On Sep 1, 2022, at 11:33, Michael Marshall wrote: > > +1 (binding) > > - Verified signatures for 45 artifacts > - Verified the checksums for 45 artifacts > - Compiled `apache-pulsar-2.8.4-src.tar.gz` running `mvn clean install > -DskipTests` using JDK 11 > - Verified `mvn apache-rat:check` passes > - Ran pulsar standalone, verified some pulsar-admin commands, verified > perf produce/consume worked using JDK 11 > > Thanks for running the release, Yunze! > > - Michael > > On Mon, Aug 29, 2022 at 10:30 AM Nicolò Boschi wrote: >> >> +1 (non binding) >> >> Checks: >> >> - Checksum and signatures >> >> - Apache Rat check passes >> >> - Compile from source >> >> - Run Pulsar standalone and produce-consume from CLI >> >> - Tested K8S installation with Datastax Pulsar helm chart and verified TLS, >> offloads and ElasticSearch sink >> >> >> >> Nicolò Boschi >> >> >> Il giorno lun 29 ago 2022 alle ore 17:20 Yunze Xu >> ha scritto: >> >>> Hi everyone, there are already two binding +1. Could anyone else help >>> verify it? >>> >>> Thanks, >>> Yunze >>> >>> >>> >>> 2022年8月23日 21:26，PengHui Li 写道： +1 (binding) - Start the standalone service - Publish and consume messages - Run the Cassandra connector - Validate the stateful function Thanks, Penghui On Tue, Aug 23, 2022 at 9:57 AM guo jiwei wrote: > +1 (binding) > > - Checked checksums and signatures > - Checked license headers using Apache Rat > - Compiled the source by JDK11 > - Ran the standalone server > - Confirmed that producer and consumer work properly > - Validated functions, connectors, and stateful functions > > > Regards > Jiwei Guo (Tboy) > > > On Mon, Aug 15, 2022 at 10:18 AM Qiang Huang >>> > wrote: > >> Got it. Thx. >> >> Yunze Xu 于2022年8月14日周日 23:22写道： >> >>> You can see >>> https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f >>> for more details about the LTS release. >>> >>> Thanks, >>> Yunze >>> >>> >>> >>> 2022年8月14日 11:00，Qiang Huang 写道： +1 (non-binding) Is 2.8.4 a long term support release? Yunze Xu 于2022年8月12日周五 16:20写道： > This is the first release candidate for Apache Pulsar, version > 2.8.4. > > It fixes the following issues: > > >>> >> > >>> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.8.4 > > *** Please download, test and vote on this release. This vote will >> stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided >> for > convenience. > > Source and binary files: > >> >>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.4-candidate-1/ > > SHA-512 checksums: > > >>> >> > >>> c3d26704f2cfb3365c29d4110612ca7351084f8bee3c306d5e906b3f9b22c7557cc5baf12f74f8c222baccae3310691419eda5b47fdf9cd6c5281b70134ab5eb > apache-pulsar-2.8.4-bin.tar.gz > >>> >> > >>> 28160ee94dccfb74dfb56e0e5d0e08870c6612659507333a52b5660ecbf060a75d1eed667cffd8596f9468de95055b78916b932db0e0d4c2745868d55429ee98 > apache-pulsar-2.8.4-src.tar.gz > > Maven staging repo: > >>> > >>> https://repository.apache.org/content/repositories/orgapachepulsar-1170/ > > The tag to be voted upon: > v2.8.4-candidate-1 (02ee5616866d4eda8dd94f85d9d9b71c459f248d) > https://github.com/apache/pulsar/releases/tag/v2.8.4-candidate-1 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Docker images: > > > >>> >> > >>> https://hub.docker.com/layers/pulsar/bewaremypower/pulsar/2.8.4/images/sha256-fba51a75c0f2ca79fbff7b254f80f641fcda661fd702f8149bbfdd5994078e3a > > > >>> >> > >>> https://hub.docker.com/layers/pulsar-all/bewaremypower/pulsar-all/2.8.4/images/sha256-42d4b41e5869edc6242bb49d6a1687bd6d191a6385637122edc5c7b2c44ee46f > > Please download the source package, and follow the Release Candidate > Validation[1] to validate the release > > [1] >> https://github.com/apache/pulsar/wiki/Release-Candidate-Validation > > Thanks, > Yunze > > > > > -- BR, Qiang Huang >>> >
[GitHub] [pulsar-client-node] equanz closed pull request #229: Update C++ client version and compatiblity table to v2.10.1
equanz closed pull request #229: Update C++ client version and compatiblity table to v2.10.1 URL: https://github.com/apache/pulsar-client-node/pull/229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: Pulsar Community Meeting Notes 2022/09/01, (8:30 AM PST)
Thank you, Michael, for doing the notes. Good job. I'd like to add a link to another email thread where there is more context to the discussion about namespace bundles. The context is necessary for interpreting the notes of the discussion. The context is in the thread "[DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface" and the email https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk . The main questions stated in the email were a starting point for the discussion yesterday: 1) Should we replace namespace bundles in Pulsar 3.0 design? 2) What is the replacement? The same thread https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk explains the context for these questions, and it would be useful to continue the dialogue in that mailing list thread. Please join the discussion! -Lari On 2022/09/02 05:39:24 Michael Marshall wrote: > Hi Pulsar Community, > > Here are the meeting notes from today's community meeting. Thanks to > all who participated! > > As you'll see, we had a very long discussion about the idea of > replacing bundles. The other two topics were on broker metadata cache > evictions and the pulsar protocol. > > Disclaimer: if something is misattributed or misrepresented, please > send a correction to this list. > > Source google doc: > https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE > > Thanks, > Michael > > 2022/09/01, (8:30 AM PST) > - Attendees: > - Matteo Merli > - Lari Hotari > - Enrico Olivelli > - Andrey Yegorov > - Dave Fisher > - Heesung Sohn > - Ayman Khalil > - Michael Marshall > - Asaf Mesika > - Christophe Bornet > > - Discussions/PIPs/PRs (Generally discussed in order they appear) > > - Lari: the mailing list discussion about namespace bundles and the > long term plan. Matteo: I would like to remove them. They can be very > annoying. They have to be big enough to be working, and small enough > to not cause imbalance problems. There is an order of magnitude > difference in the state by only managing them as groups, not as > individuals. If you have millions of topics, how can a single broker > keep track of everything? Dave: could part of the problem be that part > of the problem is that adding the broker and growing topics, you > encounter bundle splits, is the problem splitting them? Matteo: the > split is a way to maintain them. If you have a bundle with too many > topics, it is too large for even distribution of load. Dave: what if > you could allocate a bunch of bundles at once? Michael: you can do > that. Matteo: the problem there is that if you don’t need that, then > it’s a waste of resources. It’s not terrible, but worse case is when > you have one topic per bundle. If you know your use case, you can > create the namespace with the “right” number of bundles. The system is > ideally designed so that operators don’t need to configure these low > level details. Dave: is that the point of the different bundle split > algorithms, then? Matteo: yes. Lari: do we want to change the design > at any point in the future? One case is topic level anti-affinity > rules for a very high volume partitioned topic where you want to > ensure each partition is on different brokers. Matteo: it is kind of > possible today. If you want to maximize throughput, that should be the > responsibility of the load manager. It is possible to have bundle > level failure domains where bundles are split. Lari: since it is about > bundles, it is extra complexity to ensure the topics are distributed > because they could be in the same bundle depending on their hash. > Matteo: if you have 10 brokers, you should have 100 partitions if > you’re looking to spread the load across all of the brokers. Heesung: > note that there are different splitting algorithms. Michael: there is > not merging topic bundles, which makes splitting a “forever” event > right now. Matteo: merging bundles was on the road map (like 8 years > ago). It is doable, but it is complicated because it requires > coordination between multiple nodes. In most cases, it’s not terrible > to overshoot on the number of bundles. How can we support both > scenarios, (Lari’s and the millions of topics scenario)? Lari: the > design is limiting future improvements. Another use case from the > mailing list related to PIP 192 broker graceful shutdown. Due to the > way bundles are unloaded, there are interruptions for topic > availability. If you could move topics atomically, there is a chance > for a seamless handover. There could be an RPC between brokers to help > smooth over issues, too. Matteo: that implies every broker knows about > every topic. Lari: no, that isn’t my proposed design. Matteo: if you > have ownership, then brokers need to know. In the work for 192, there > is a concept of transfer. I wouldn’t want RPCs because it could become > tricky with availability. Part of the PIP is to keep track of bundle > placeme
Re: [DISCUSS] Call to improve Pulsar contributor's experience
Thank you Lari! Except for updating .md docs, we need to update various API docs [1]. These docs are generated from code automatically (annotations in code files). For these special doc PRs, can we set them to run only build (compile) tests and skip other code tests? ~~ [1] Admin API docs: - pulsar-admin: update https://github.com/apache/pulsar/tree/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli - REST API: update https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar - Java admin API: update https://github.com/apache/pulsar/tree/master/pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin Client API docs: - Java: update https://github.com/apache/pulsar/tree/master/pulsar-client-api/src/main/java/org/apache/pulsar/client/api - CPP: update https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/include/pulsar - Python: have no idea ~~ Thank you! Yu On Fri, Sep 2, 2022 at 1:17 AM Lari Hotari wrote: > On 2022/09/01 08:36:11 Yu wrote: > > # 1 > > For pure doc PRs (only update .md files), do they run the same tests as > > code PRs? > > If so, can we set them to run only doc-related tests and skip code tests > > (since they're easily failed)? > > In this way, docs can be iterated faster. > > The solution is already in place where the CI pipeline for docs is > expedited. > Some technical details about the solution: All builds steps in the GitHub > Actions workflow build jobs are skipped for PRs that include changes only > to docs. The reason why the workflows and build jobs aren't completely > skipped is that we use the "required checks" feature and it is necessary to > run all required checks also for PRs with only doc changes. > > > > > > > > > # 2 > > Does it make sense to add instructions for tests to the Pulsar > Contribution > > Guide? > > > > For example, > > > > * For users: > > - How to resolve test issues (common test failure reasons and solutions) > > - Who can ask for help if users are blocked and can not resolve problems > > themselves > > - How to report test bugs > > > > * For developers: > > - How do tests work? (mechanism, Apache rules, etc) > > - How can I add/update tests? (quotas [1], limitations, notes, etc) > > Good suggestions. > > In general, I hope we find better ways to listen to the voice of our > contributors. What is their contribution experience? How did they feel > about it? > > Perhaps we could decide to conduct surveys? GitHub discussions has support > for polls [1] so that is one option as a technical solution for asking for > feedback in a way where there would be a low barrier to respond. > > -Lari > > [1] https://github.blog/changelog/2022-04-12-discussions-polls/ >
[GitHub] [pulsar-client-node] equanz commented on pull request #229: Update C++ client version and compatiblity table to v2.10.1
equanz commented on PR #229:
URL:
https://github.com/apache/pulsar-client-node/pull/229#issuecomment-1235257092
In `apachepulsar/pulsar-build:ubuntu-16.04` image, use
`-Xlog:gc*:logs/pulsar_gc_%p.log:time,uptime:filecount=10,filesize=20M` option
for runtime.
https://github.com/apache/pulsar/blob/v2.10.1/conf/pulsar_env.sh#L50-L58
However, pulsar command uses
`JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64` in runtime.
https://hub.docker.com/layers/apachepulsar/pulsar-build/ubuntu-16.04/images/sha256-57174595e685002c0e4ce768361af8c10fcf30310f466966c3f8663a8638e4b7?context=explore
```
Unrecognized option:
-Xlog:gc*:logs/pulsar_gc_%p.log:time,uptime:filecount=10,filesize=20M
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
```
(The script file was fixed in https://github.com/apache/pulsar/pull/14711 .)
I updated the docker image tag to `ubuntu-20.04` to avoid this issue.
`JAVA_HOME` was not set in `ubuntu-20.04` image.
https://hub.docker.com/layers/apachepulsar/pulsar-build/ubuntu-20.04/images/sha256-5f50d029fa6ba5a1d410faf7d4d0539200246572a1459cb6e3cc32219300f591?context=explore
Also, I modified test scripts to fix the below issue.
```
fatal: unsafe repository ('/pulsar-client-node' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /pulsar-client-node
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: Pulsar CI congested, master branch build broken
Lari,
Il giorno mar 30 ago 2022 alle ore 14:39 Lari Hotari
ha scritto:
>
> Pulsar CI continues to be congested, and the build queue is long.
>
> I would strongly advice everyone to use "personal CI" to mitigate the issue
> of the long delay of CI feedback. You can simply open a PR to your own
> personal fork of apache/pulsar to run the builds in your "personal CI".
> There's more details in the previous email in this thread.
>
> Some updates:
>
> There has been a discussion with Gavin McDonald from ASF infra on the-asf
> slack about getting usage reports from GitHub to support the investigation.
> Slack thread is the same one mentioned in the previous email,
> https://the-asf.slack.com/archives/CBX4TSBQ8/p1661512133913279 . Gavin
> already requested the usage report in GitHub UI, but it produced invalid
> results.
>
> I made a change to mitigate a source of additional GitHub Actions overhead.
> In the past, each cherry-picked commit to a maintenance branch of Pulsar has
> triggered a lot of workflow runs.
>
> The solution for cancelling duplicate builds automatically is to add this
> definition to the workflow definition:
> concurrency:
> group: ${{ github.workflow }}-${{ github.ref }}
> cancel-in-progress: true
>
> I added this to all maintenance branch GitHub Actions workflows:
>
> branch-2.10 change:
> https://github.com/apache/pulsar/commit/5d2c9851f4f4d70bfe74b1e683a41c5a040a6ca7
> branch-2.9 change:
> https://github.com/apache/pulsar/commit/3ea124924fecf636cc105de75c62b3a99050847b
> branch-2.8 change:
> https://github.com/apache/pulsar/commit/48187bb5d95e581f8322a019b61d986e18a31e54
> branch-2.7:
> https://github.com/apache/pulsar/commit/744b62c99344724eacdbe97c881311869d67f630
>
> branch-2.11 already contains the necessary config for cancelling duplicate
> builds.
>
> The benefit of the above change is that when multiple commits are
> cherry-picked to a branch at once, only the build of the last commit will get
> run eventually. The builds for the intermediate commits will get cancelled.
> Obviously there's a tradeoff here that we don't get the information if one of
> the earlier commits breaks the build. It's the cost that we need to pay.
> Nevertheless our build is so flaky that it's hard to determine whether a
> failed build result is only caused by bad flaky test or whether it's an
> actual failure. Because of this we don't lose anything by cancelling builds.
> It's more important to save build resources. In the maintenance branches for
> 2.10 and older, the average total build time consumed is around 20 hours
> which is a lot.
Thanks !
Enrico
>
> At this time, the overhead of maintenance branch builds doesn't seem to be
> the source of the problems. There must be some other issue which is possibly
> related to exceeding a usage quota. Hopefully we get the CI slowness issue
> solved asap.
>
> BR,
>
> Lari
>
>
> On 2022/08/26 12:00:20 Lari Hotari wrote:
> > Hi,
> >
> > GitHub Actions builds have been piling up in the build queue in the last
> > few days.
> > I posted on bui...@apache.org
> > https://lists.apache.org/thread/6lbqr0f6mqt9s8ggollp5kj2nv7rlo9s and
> > created INFRA ticket https://issues.apache.org/jira/browse/INFRA-23633
> > about this issue.
> > There's also a thread on the-asf slack,
> > https://the-asf.slack.com/archives/CBX4TSBQ8/p1661512133913279 .
> >
> > It seems that our build queue is finally getting picked up, but it would be
> > great to see if we hit quota and whether that is the cause of pauses.
> >
> > Another issue is that the master branch broke after merging 2 conflicting
> > PRs.
> > The fix is in https://github.com/apache/pulsar/pull/17300 .
> >
> > Merging PRs will be slow until we have these 2 problems solved and existing
> > PRs rebased over the changes. Let's prioritize merging #17300 before
> > pushing more changes.
> >
> > I'd like to point out that a good way to get build feedback before sending
> > a PR, is to run builds on your personal GitHub Actions CI. The benefit of
> > this is that it doesn't consume the shared quota and builds usually start
> > instantly.
> > There are instructions in the contributors guide about this.
> > https://pulsar.apache.org/contributing/#ci-testing-in-your-fork
> > You simply open PRs to your own fork of apache/pulsar to run builds on your
> > personal GitHub Actions CI.
> >
> > BR,
> >
> > Lari
> >
> >
> >
> >
> >
> >
> >
> >
[GitHub] [pulsar-manager] rdhabalia commented on a diff in pull request #486: Adding https support for backend service
rdhabalia commented on code in PR #486:
URL: https://github.com/apache/pulsar-manager/pull/486#discussion_r961978656
##
src/main/java/org/apache/pulsar/manager/service/impl/BrokerStatsServiceImpl.java:
##
@@ -124,13 +129,22 @@ private void scheduleCollectStats() {
clusterLists.forEach((clusterMap) -> {
String cluster = (String) clusterMap.get("cluster");
Pair envCluster = Pair.of(env.getName(),
cluster);
-String webServiceUrl = (String) clusterMap.get("serviceUrl");
+
+String webServiceUrl = (String) (tlsEnabled &&
clusterMap.containsKey("serviceUrlTls") && StringUtils.isNotBlank((String)
clusterMap.get("serviceUrlTls")) ?
Review Comment:
```
String serviceUrlTls = clusterMap.get("serviceUrlTls");
tlsEnabled = tlsEnabled && StringUtils.isNotBlank(serviceUrlTls);
String webServiceUrl = tlsEnabled ? serviceUrlTls :
clusterMap.get("serviceUrl");
at line#140
if(!url.startsWith("http") {
url = (tlsEnabled ? "https://"; : "http://";) + url;
}
we can remove line#144
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
CloudEvents binding requires standard for Pulsar
Hi recently discovered the discussion around creating a CloudEvent binding for Pulsar. https://github.com/cloudevents/spec/pull/237 It appears that Pulsar doesn't meet their minimum requirements due to lack of a standard protocol. See https://github.com/cloudevents/spec/pull/254 Their comment says: "For a protocol or encoding to qualify for a core CloudEvents event format or protocol binding, it must belong to either one of the following categories: - The protocol has a formal status as a standard with a widely-recognized multi-vendor protocol standardization body (e.g. W3C, IETF, OASIS, ISO) - The protocol has a "de-facto standard" status for its ecosystem category which means it is used so widely that it is considered a standard for a given application. Practically, we would like to see at least one open source implementation and at least a dozen independent vendors using it in their products/services. " As CloudEvents is gaining momentum within CNCF, this may become a problem. Has their been any discussion around standardization and how we might meet this requirement? -- Devin Bost Sent from mobile Cell: 801-400-4602
Re: [DISCUSS] Apache Pulsar 2.10.2 release
+1, thanks for volunteering to do another release, Haiting! Thanks, Michael On Thu, Aug 25, 2022 at 7:43 AM Anon Hxy wrote: > > +1 > > Thanks, > Xiaoyu Hou > > Haiting Jiang 于2022年8月23日周二 22:10写道： > > > Hello, Pulsar community: > > > > I'd like to propose to release Apache Pulsar 2.10.2 > > > > Currently, we have 189 commits [0] labeled with `release/2.10.2`, > > And 39 of them are not cherry-picked yet [1]. > > > > And there are 39 open PRs [2], I will follow them to make sure that > > the important fixes could be contained in 2.10.2. > > > > If you have any important fixes or any questions, > > please reply to this email, we will evaluate whether to > > include it in 2.10.2 > > > > [0] > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+ > > [1] > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+-label%3Acherry-picked%2Fbranch-2.10+ > > [2] > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.2+ > > > > Best Regards > > Haiting Jiang > >
[GitHub] [pulsar-site] tisonkun opened a new pull request, #197: Set notifications and merge buttons
tisonkun opened a new pull request, #197: URL: https://github.com/apache/pulsar-site/pull/197 cc @dave2wave @urfreespace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
