Re: [DISCUSS] PIP-280 : Refactor CLI Argument Parsing Logic for Measurement Units using JCommander's custom converter
I read the proposal on GitHub and it's clear to reduce code duplication with a middle size refactor. Although I think it may not require a PIP, we can try to proceed in the PIP process and see if the PIP process fits this proposal. cc developers in this domain AFAIK. Best, tison. Joo Hyuk Kim 于2023年6月30日周五 15:20写道： > Hi, community > > > I am proposing to refactor `Cmd*` classes’s measurement parsing logic. > > > ## Background > > > As it stands, the parsing logic for measurement units such as time and > bytes in various CLI classes is implemented verbosely, contained in the Cmd > converter themselves. Leading to a lack of code reuse and possible > inconsistencies. > > > ## Proposal > > > The idea is to refactor all `Cmd` classes to utilize the converter > functionality of JCommander (link[1]). This will allow us to isolate the > measurement-specific parsing logic and streamline the code, thereby > improving maintainability and reusability. See the concrete example in this > PIP for a more in-depth explanation. > > > There is ongoing PR to provide concept in (link [2]) > > > ## Concerns > > > 1. Will this create too much work or introduce confusion? > > > By working class-by-class or inner-class basis, we can gradually refactor > the codebase without causing disruption. The changes are expected to > increase readability and maintainability, making future modifications > easier and less error-prone. > > > 2. Will it affect existing functionality? > > > The proposed changes are meant to streamline and standardize the code > without altering functionality. > > > ## Implementation > > > The initial focus will be on the following classes: > > > - `CmdNamespaces.java` > > - `CmdTopics.java` > > - `CmdTopicPolicies.java` > > > More classes may be added as the refactoring process continues. > > > Let me know what you think. > > > Best, > > JooHyuk > > > PIP : https://github.com/apache/pulsar/pull/20691 > > > Links : > > [1] https://jcommander.org/#_custom_types_converters_and_splitters > > [2] https://github.com/apache/pulsar/pull/20663 >
[VOTE] Pulsar Client Go Release 0.11.0 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 0.11.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Go client, version 0.11.0. It fixes the following issues: https://github.com/apache/pulsar-client-go/milestone/13?closed=1 Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1048 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.0-candidate-1/ The tag to be voted upon: v0.11.0 https://github.com/apache/pulsar-client-go/tree/v0.11.0-candidate-1 SHA-512 checksums: 72cf0f71c82095f0d412ff15b35e32c56f118ec7f297ad662073c14786fead0a416d7b1a8b8caa890b76bd722cba1734e52fd4f53ee3daa793cb5a7915e5fb32 apache-pulsar-client-go-0.11.0-src.tar.gz
Re: [DISCUSS] Apache Pulsar 2.11.2 release
Hi Enrico, Which BK release do we want to wait? I noticed that the BookKeeper community is releasing 4.14.8 and 4.16.2, but Pulsar 2.11 is using 4.15. Regards, Penghui On Mon, Jun 26, 2023 at 2:56 PM Enrico Olivelli wrote: > Jiwei, > > Il giorno lun 26 giu 2023 alle ore 08:09 guo jiwei > ha scritto: > > > > Hello, community: > >It has been more than 2 months since the release of 2.11.1. During > this > > period, we have more than 80 fixes. > > https://github.com/apache/pulsar/compare/v2.11.1...branch-2.11 > >I suggest releasing 2.11.2. If you have no comments, I will check the > > existing PR that needs to cherry-pick to the 2.11 branch. > >If you have a PR that needs to cherry-pick, please leave a message > here. > > > Probably we want to wait for a BK release, apart from that I am +1 > > Enrico > > > > > Regards > > Jiwei Guo (Tboy) >
Re: [DISCUSS] Pluggable Pulsar Functions runtime to support new runtimes
Hi, Lari. Thanks for your proposal. Sounds good to me. Supporting function pluggable essentially decouples the scheduling layer and the computing layer. This can not only extend the function runtime in more languages but even use this feature to implement a new non-function runtime plugin, such as the EventSouring development model. Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. At present, the scheduling logic of the function will not be related to the key of the message, I understand that it belongs to the internal feature of the runtime, and this can also be enhanced based on the current runtime? I don't quite understand what this has to do with runtime pluginization. Can you explain? In addition, the Pulsar Functions have a missing piece in how functions are > mapped to instances. It's not very efficient to even run each and every > function as a separate deployable entity. The cost of each independent JVMs > is high. It would be also better to have a model where where could be a > group of functions that are provided by one instance and always run > together. Having this option could bring down the cost and also improve the > developer experience. The framework shouldn't require the developer that > each individual function is deployed in a separate .jar file which gets run > in a separate JVM. This means that we need to enhance the function interface to that the user tells the runtime layer which topic the function subscription cares about. In addition, if we want to implement concurrent sequential processing based on keys, we can also let the function register the key of interest. These can be implemented in the current runtime, which has nothing to do with plug-in runtime, right? I wonder what are the future plans after plugging in the runtime? Do you want to develop a new runtime that implements many of the benefits you enumerate, or will you continue to enhance it on the original runtime (Java). Thanks, Baodi Shi On Jun 20, 2023 at 15:33:22, Lari Hotari wrote: > Dear Pulsar Community Members, > > I would like to initiate a discussion on making the Pulsar Functions > runtime "pluggable". In doing so, we can ensure that the addition of new > runtime types becomes more straightforward. > > This use case will allow us to add support for Pulsar Functions based on > various platforms such as: > > * Pulsar Client Reactive > * Node.js / JavaScript > * WebAssembly (WASM) > * Spring Pulsar & Reactive Spring > > One of the weak points in the current Pulsar Functions runtime is the > default handling of messages individually. Individual message processing > can be slow and inefficient in cases where the main function of the > Pulsar Function (or Sink) is to do backend API calls. > > Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. > > The Reactive Pulsar Client provides an inbuilt solution for implementing > pipelining. With its ReactiveMessagePipelineBuilder, we can configure > concurrency levels with key-ordered processing support. This capability > could potentially eliminate the need to use key-shared subscriptions to > scale Pulsar processing. If a reactive Pulsar Function were primarily to > serve as a router for API calls, we could adjust the concurrency level > to hundreds or even thousands, provided the backend could handle the > load. > > With a pluggable Pulsar Functions runtime, we could introduce new > runtime types without the need for implementing each type in the > upstream project. This strategy could likely lead to new opportunities > for innovative ideas and contributions in this field. > > I am interested to know your thoughts on making the Pulsar Functions > runtime pluggable so that we can add new runtime types. > > Best Regards, > > -Lari >
