Besides, this proposal is a bit different from other PIPs. A normal workflow for a PIP should be: 1. Discuss and vote for the proposal that contains high level design (it requires 3 binding +1s from PMC members) 2. Review the PR that implements the design in details (it requires a successful CI workflow and at least 1 approval from committers)
However, for such a PIP, it's confusing for how we should make progress? Opening an empty repo and allowing users to push commits? Or moving an existing repo to Apache directly? The 1st solution might be good so all commits can be reviewed by committers. What I have concern is that each commit requires approvals from Pulsar committers and it's less convenient than maintaining it by individuals. For the 2nd solution, we must need an official review process on it, not just by the PIP vote. Thanks, Yunze On Tue, Aug 13, 2024 at 5:02 PM Yunze Xu <x...@apache.org> wrote: > > Hi Dave, > > We had some offline discussions with the author last week about this > repo. So let me clarify some points here. > > First, I voted -1 against making > https://github.com/StevenLuMT/pulsar-java-contrib managed by Apache. > As you can see, there are no licenses, no tests, no test workflows, no > release plans, and no more commits in the last two weeks [2]. > Excluding the best practice parts, there is only one example for now, > without any test or documentation. > > Secondly, maintaining this repo requires Pulsar PMC's care. Otherwise, > what's the advantage of having it maintained by Apache? Is there any > PMC member that is willing to maintain it? > > I'm +1 to have such a contributor repository. But for now, maintaining > it in the author's own repo should be easier to maintain. We can link > it in our Apache repo to make more people know this repo. > > If this repo was taken good care of and used by more users, I would be > glad to have it in Apache. But at least not for now. > > [1] > https://github.com/StevenLuMT/pulsar-java-contrib/tree/stable/pulsar-client-common-contrib/src/main/java/org/apache/pulsar/client/api/impl > [2] https://github.com/StevenLuMT/pulsar-java-contrib/commits/stable/ > > Thanks, > Yunze > > On Tue, Aug 13, 2024 at 12:34 AM Dave Fisher <w...@apache.org> wrote: > > > > Hi Yunze, > > > > A few of your points are of concern and to me could be good arguments for > > this extra code repository. > > > > On 2024/08/05 13:25:24 Yunze Xu wrote: > > > Let's ignore the last sentence I mentioned. I mentioned my suspicion > > > because I have concerns about the lifetime of this repo. It's > > > clarified privately so I'm apologizing for my conjecture about the > > > company. So let's focus on the topic itself. > > > > Thank you what our employers are doing with or without Pulsar must be of no > > concern to the merits of each of us who are here as INDIVIDUALS. > > > > > > > > > > I just thought again about the "best practice" part. Why not > > > > contribute it to the Apache official website > > > > (https://github.com/apache/pulsar-site)? > > > > One way to use this repository is to develop code examples and best > > practices in a shared environment in public. > > > > > > > > > > > > TL;DR, such a repository is not necessary to be contributed to Apache. > > > > > > > > > > Actually, only the "Collect user best practices" point makes sense to > > > > > me. The reason to have so many pluggables is exactly to avoid the core > > > > > repo being not so bloated. > > > > > > > > > > Yeah, OTel is a good example. But IMHO, Pulsar is far less populated > > > > > than OTel. In the current phase, it's more important to improve the > > > > > code quality of such plugins, including > > > > > - Metadata store. From my personal perspective, I don't suggest making > > > > > InMemory, Etcd, Oxia as the built-in implementations. Having ZK as > > > > > default and RocksDB for a lightweight standalone is enough. > > > > True, but these other metadata stores are being used and developing > > examples in public can only help others understand how to make use of the > > many years effort to enable these other metadata stores without making > > these fully builtin. Etcd is a project that has very wide use and it may be > > good for Pulsar to share how to use it. > > > > Apache JClouds is going to the Apache Attic soon. Pulsar depends on it. > > Perhaps this repository could allow development of an alternative approach > > in a pre-PIP don’t break the CI manner. > > > > > > > - Load Manager. If you have the demand to implement a customized load > > > > > manager (like me), you will understand it. The extensible load manager > > > > > introduced from PIP-192 makes things better. But it's still a mess. > > > > > See how many `isLoadManagerExtensionEnabled` calls in the repo. > > > > > - Schema Registry. It's not pluggable yet. But I've heard many users > > > > > want a more powerful implementation. > > > > > > > > > > These three typical plugins above are really not pluggable enough. > > > > > Before adding them to such a repository, it's more worth taking some > > > > > time to revisit the implementation and make them better. Just like the > > > > > producer and consumer interceptors, if some implementations are > > > > > general enough, we should accept them in the core repo or create > > > > > another repo for them. > > > > > > > > > > Besides, these pluggables provide a way for companies to implement its > > > > > own private plugins as the sell points. For example, I really don't > > > > > think my employer will be glad to contribute ideas of our internal > > > > > plugins to this repo. > > > > Your ability to contribute is defined by a combination of the ICLA that you > > signed and your work contract with your employer. If you cannot contribute > > then please do not. If others have no such restriction then it is to the > > benefit of the Pulsar community for them to be enabled. > > > > > > > > > > > > > > So. It's -1 for me. But I think it's good to incubate this repo for > > > > > some time and see if there are many people willing to contribute, > > > > > rather than the XHS team itself. > > > > I do not see ‘XHS’, I see an individual who has a good idea about how to > > become more involved in contributing to Pulsar in a more incremental manner. > > > > > > > > > > > > BTW, I'm also curious about the position of Pulsar in XHS since I see > > > > > your company is adopting AutoMQ recently. I'm suspicious that > > > > > contributing such a repo into Apache is a KPI in the internal > > > > > competition. > > > > Best, > > Dave > > > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > On Sat, Aug 3, 2024 at 5:37 PM xiangying meng <xiangy...@apache.org> > > > > > wrote: > > > > > > > > > > > > Dear all, > > > > > > > > > > > > I would like to express my appreciation to all of you who have shown > > > > > > support for my recent proposal. Your feedback and encouragement are > > > > > > greatly appreciated and have been instrumental in moving the > > > > > > discussion forward. > > > > > > > > > > > > I am pleased to announce that I have initiated a vote on this > > > > > > proposal. You can find the details and participate in the voting > > > > > > through the following link: > > > > > > > > > > > > https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y > > > > > > > > > > > > Your active participation is crucial for the success of this > > > > > > initiative, and I look forward to a positive outcome. > > > > > > > > > > > > Once again, thank you for your valuable input and support. > > > > > > > > > > > > Best regards, > > > > > > Xiangying > > > > > > > > > > > > On Sat, Aug 3, 2024 at 3:38 AM Dave Fisher <w...@apache.org> wrote: > > > > > > > > > > > > > > Hi - I renamed the PIP PR to include the PIP number - PIP-367. - > > > > > > > https://github.com/apache/pulsar/pull/23061 > > > > > > > > > > > > > > I think that you have covered concerns and it is time to start a > > > > > > > separate VOTE thread. > > > > > > > > > > > > > > Assuming that the PIP passes then we can create the new > > > > > > > repository by following your example and adjust the files to best > > > > > > > fit policies. > > > > > > > > > > > > > > Best Regards, > > > > > > > Dave > > > > > > > > > > > > > > > On Jul 29, 2024, at 7:45 PM, xiangying meng > > > > > > > > <xiangy...@apache.org> wrote: > > > > > > > > > > > > > > > > Dear Dave and Enrico, > > > > > > > > > > > > > > > > Thank you for your reply and valuable comments. I think the > > > > > > > > point that > > > > > > > > needs to be clarified now is whether this repository needs to be > > > > > > > > released. In the original idea, the functions in this > > > > > > > > repository are > > > > > > > > not stable. It will be used to collect various experimental > > > > > > > > functions > > > > > > > > and extended plug-in implementations. > > > > > > > > The risk is mentioned in the first version of the proposal. > > > > > > > > However, > > > > > > > > due to lack of experience, I failed to clarify the importance of > > > > > > > > release, so there is no clear release description. Thanks to > > > > > > > > Dave for > > > > > > > > helping to further clarify this matter. > > > > > > > > > > > > > > > > - For experimental functions and customized functions that > > > > > > > > require > > > > > > > > modifying the Pulsar source code, I think they must not be > > > > > > > > released. > > > > > > > > - For the plug-in implementation of Pulsar extension, it cannot > > > > > > > > be > > > > > > > > promised to users that it is stable, so it cannot be released, > > > > > > > > too. > > > > > > > > > > > > > > > > Therefore, we further clarify the positioning of this > > > > > > > > repository. > > > > > > > > > > > > > > > > - Collect user needs and their various customized plug-in > > > > > > > > implementations > > > > > > > > - Collect and organize various experimental functions and > > > > > > > > customized functions > > > > > > > > - Collect user best practices > > > > > > > > > > > > > > > > The admission standard of the warehouse is that this function > > > > > > > > can work > > > > > > > > normally and meet the needs, but it is not necessarily stable, > > > > > > > > so it > > > > > > > > will not be released. > > > > > > > > > > > > > > > > With the continuous feedback from users, this feature may become > > > > > > > > stable, and eventually someone (new contributor or original > > > > > > > > contributor) may move this feature to a stable releaseable > > > > > > > > place. That > > > > > > > > is, move it to the `appropriate Pulsar repository and > > > > > > > > component` you > > > > > > > > mentioned. > > > > > > > > > > > > > > > > In addition, regarding the SECURITY.md file, I have added it to > > > > > > > > the > > > > > > > > repository. [0] > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > xiangying > > > > > > > > > > > > > > > > [0] - > > > > > > > > https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/SECURITY.md > > > > > > > > > > > > > > > > On Mon, Jul 29, 2024 at 10:50 PM Dave Fisher <w...@apache.org> > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >>> On Jul 29, 2024, at 6:37 AM, Enrico Olivelli > > > > > > > >>> <eolive...@gmail.com> wrote: > > > > > > > >>> > > > > > > > >>> Il giorno lun 29 lug 2024 alle ore 11:55 steven lu > > > > > > > >>> <lushiji2...@gmail.com> > > > > > > > >>> ha scritto: > > > > > > > >>> > > > > > > > >>>> Different from pulsar-adapters, we added some instructions > > > > > > > >>>> and examples > > > > > > > >>>> here https://github.com/StevenLuMT/pulsar-java-contrib > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> This example is clear, thanks ! > > > > > > > >> > > > > > > > >> From your README in the repository. > > > > > > > >> > > > > > > > >>> Pulsar java contrib is to provide a non-core code maintenance > > > > > > > >>> repository to collect plugin implementations, personalized > > > > > > > >>> features, experimental features, and best practices from > > > > > > > >>> users. > > > > > > > >> > > > > > > > >> These are examples which anyone can use at their own risk and > > > > > > > >> the PMC does not intend to make RELEASES. Is this correct? If > > > > > > > >> so then that should be very clear. > > > > > > > >> > > > > > > > >> Anything the community decides to support for the future in a > > > > > > > >> release should be moved into the appropriate Pulsar repository > > > > > > > >> and component. > > > > > > > >> > > > > > > > >> If this were made clear then I would be a +1. > > > > > > > >> > > > > > > > >> Security vulnerabilities found would still require care and > > > > > > > >> attention from the PMC due to the private nature of this > > > > > > > >> reporting. You will need a SECURITY.md file. > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Dave > > > > > > > >> > > > > > > > >>> > > > > > > > >>> Let's put it this way: +0 > > > > > > > >>> I am not against this proposal, but I would support it if we > > > > > > > >>> see that the > > > > > > > >>> Community and in particular the PMC > > > > > > > >>> is willing to maintain it. > > > > > > > >>> > > > > > > > >>> If the community is not willing to maintain it (and we > > > > > > > >>> already have > > > > > > > >>> problems with pulsar-adapters for instance) then > > > > > > > >>> it is only like adding another dead repository plus the > > > > > > > >>> responsibility to > > > > > > > >>> fix security issues and keep the examples up to date with the > > > > > > > >>> best > > > > > > > >>> practices as long as Pulsar evolves > > > > > > > >>> > > > > > > > >>> If I were you I would start a well curated github repository > > > > > > > >>> and encourage > > > > > > > >>> people to contribute their code snippets to it. > > > > > > > >>> > > > > > > > >>> That said, I won't VOTE against the proposal > > > > > > > >>> > > > > > > > >>> Thanks > > > > > > > >>> > > > > > > > >>> Enrico > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> Enrico Olivelli <eolive...@gmail.com> 于2024年7月23日周二 17:45写道： > > > > > > > >>>> > > > > > > > >>>>> Probably this repository is already what you want to do: > > > > > > > >>>>> https://github.com/apache/pulsar-adapters > > > > > > > >>>>> > > > > > > > >>>>> It contains a lot of different stuff that we don't keep in > > > > > > > >>>>> the main > > > > > > > >>>> pulsar > > > > > > > >>>>> repository. > > > > > > > >>>>> > > > > > > > >>>>> Enrico > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > >>>>> Il giorno mar 23 lug 2024 alle ore 09:51 Jia Zhai > > > > > > > >>>>> <zhai...@apache.org> > > > > > > > >>>> ha > > > > > > > >>>>> scritto: > > > > > > > >>>>> > > > > > > > >>>>>> Thanks for the proposal. It is a good idea. It is also not > > > > > > > >>>>>> a bad idea > > > > > > > >>>> to > > > > > > > >>>>>> incubate it outside of Pulsar official repository at first. > > > > > > > >>>>>> > > > > > > > >>>>>> On Tue, Jul 23, 2024 at 2:59 PM Aurora Twinkle < > > > > > > > >>>> foreverlove...@gmail.com > > > > > > > >>>>>> > > > > > > > >>>>>> wrote: > > > > > > > >>>>>> > > > > > > > >>>>>>> Hi: > > > > > > > >>>>>>> I think this is a very good idea. > > > > > > > >>>>>>> > > > > > > > >>>>>>> In the native pulsar warehouse, there are some interfaces > > > > > > > >>>>>>> that have > > > > > > > >>>> no > > > > > > > >>>>>>> default implementation classes, such as > > > > > > > >>>>>>> ConsumerInterceptor, > > > > > > > >>>>>>> ProducerInterceptor. Users must implement the interface > > > > > > > >>>>>>> themselves, > > > > > > > >>>>>>> which increases the cost of use. If there is a plug-in > > > > > > > >>>>>>> library > > > > > > > >>>>>>> mentioned in this PIP, we can provide some common > > > > > > > >>>>>>> interface > > > > > > > >>>>>>> implementations in the plug-in library, such as mertics > > > > > > > >>>>>>> collection > > > > > > > >>>>>>> interceptor for producer and consumer. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Best Regards, > > > > > > > >>>>>>> AuroraTwinkle > > > > > > > >>>>>>> > > > > > > > >>>>>>> xiangying meng <xiangyingme...@gmail.com> 于2024年7月23日周二 > > > > > > > >>>>>>> 11:25写道： > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Hello Wen Zhi, > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> This project is a new repository similar to a connector, > > > > > > > >>>>>>>> and it > > > > > > > >>>> will > > > > > > > >>>>>>>> not make any modifications to Pulsar itself. [0] > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Its code part is simply a plugin library, based on > > > > > > > >>>>>>>> Pulsar's > > > > > > > >>>> external > > > > > > > >>>>>>>> interfaces, providing a rich set of implementations. It > > > > > > > >>>>>>>> only > > > > > > > >>>> retains > > > > > > > >>>>> a > > > > > > > >>>>>>>> "curated list", which is a series of pointers to > > > > > > > >>>>>>>> personalized > > > > > > > >>>>> features > > > > > > > >>>>>>>> and experimental features in personal repositories for > > > > > > > >>>>>>>> reference > > > > > > > >>>> and > > > > > > > >>>>>>>> inspiration. If these experimental features are > > > > > > > >>>>>>>> recognized by other > > > > > > > >>>>>>>> users, they can also be gradually merged into the Pulsar > > > > > > > >>>>>>>> main > > > > > > > >>>>>>>> repository. However, it will not fork the content of the > > > > > > > >>>>>>>> Pulsar > > > > > > > >>>>>>>> repository for modification in the new repository. > > > > > > > >>>>>>>> This is the improvement we made after discussion in the > > > > > > > >>>>>>>> last > > > > > > > >>>> email. I > > > > > > > >>>>>>>> will synchronize it to the proposal later. > > > > > > > >>>>>>>> Finally, thank you for your valuable suggestions, and we > > > > > > > >>>>>>>> look > > > > > > > >>>> forward > > > > > > > >>>>>>>> to your continued participation. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> [0] - https://github.com/apache/pulsar-connectors > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Best Regards, > > > > > > > >>>>>>>> Xiangying > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> On Tue, Jul 23, 2024 at 10:36 AM WenZhi Feng < > > > > > > > >>>> thetumb...@apache.org> > > > > > > > >>>>>>> wrote: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Hi, xiangying, > > > > > > > >>>>>>>>> First of all, thanks for driving the proposal. > > > > > > > >>>>>>>>> I have some questions. > > > > > > > >>>>>>>>> 1. Will this repository maintained by committers/pmc? > > > > > > > >>>>>>>>> If the > > > > > > > >>>>> answer > > > > > > > >>>>>>> is yes, there will be too many new modules introduced > > > > > > > >>>>>>> into pulsar and > > > > > > > >>>>>> hard > > > > > > > >>>>>>> to maintain. If the answer is no, i think we have better > > > > > > > >>>>>>> not to > > > > > > > >>>> include > > > > > > > >>>>>> it > > > > > > > >>>>>>> into pulsar official repository. > > > > > > > >>>>>>>>> 2. For those plugin-in feature, we may create a new > > > > > > > >>>>>>>>> repository > > > > > > > >>>>> like > > > > > > > >>>>>>> various connectors for pulsar. But for those core code > > > > > > > >>>>>>> change on > > > > > > > >>>>> pulsar, > > > > > > > >>>>>>> how can we seperate it with pulsar repository? > > > > > > > >>>>>>>>> It is thrilling that users contribute various new > > > > > > > >>>>>>>>> features to > > > > > > > >>>>>> apache > > > > > > > >>>>>>> community, enriching the ecosystem. I think we should > > > > > > > >>>>>>> find out a good > > > > > > > >>>>> way > > > > > > > >>>>>>> to realize it. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> Thanks, > > > > > > > >>>>>>>>> Wenzhi Feng. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> On 2024/07/22 17:40:24 xiangying meng wrote: > > > > > > > >>>>>>>>>> Thank you for your feedback, let's further clarify > > > > > > > >>>>>>>>>> this matter. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> The original design of this proposal is a new > > > > > > > >>>>>>>>>> contributor > > > > > > > >>>>>> repository, > > > > > > > >>>>>>>>>> and provides two ways to contribute. > > > > > > > >>>>>>>>>> The first one: > > > > > > > >>>>>>>>>> Contributors contribute through PR and accept code > > > > > > > >>>>>>>>>> review, and > > > > > > > >>>>> can > > > > > > > >>>>>>>>>> only be merged into the main branch after reviewed. > > > > > > > >>>>>>>>>> The code > > > > > > > >>>>> merged > > > > > > > >>>>>>> in > > > > > > > >>>>>>>>>> this way is various plug-ins based on Pulsar's external > > > > > > > >>>>> interface. > > > > > > > >>>>>> It > > > > > > > >>>>>>>>>> does not contain any pulsar code, only Pulsar's > > > > > > > >>>>>>>>>> dependencies. > > > > > > > >>>>>>>>>> The second one: > > > > > > > >>>>>>>>>> Features that require modifications to the Pulsar main > > > > > > > >>>> repository > > > > > > > >>>>>>> code > > > > > > > >>>>>>>>>> and may conflict with later versions. My initial idea > > > > > > > >>>>>>>>>> was for > > > > > > > >>>>>>>>>> contributors to create branches and then write the > > > > > > > >>>>>>>>>> commit id > > > > > > > >>>> and > > > > > > > >>>>>>>>>> features description into the document after the PR is > > > > > > > >>>> reviewed. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> However, after the discussion just now, I agree that > > > > > > > >>>>>>>>>> the second > > > > > > > >>>>> one > > > > > > > >>>>>>>>>> may not be maintained in the branch of the new > > > > > > > >>>>>>>>>> repository, but > > > > > > > >>>> in > > > > > > > >>>>>> the > > > > > > > >>>>>>>>>> contributor's personal repository. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Based on the above content, let's continue to look at > > > > > > > >>>>>>>>>> the > > > > > > > >>>>> question > > > > > > > >>>>>>> you raised. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>>> - How can the Pulsar community maintain a fork of > > > > > > > >>>>>>>>>>> Pulsar > > > > > > > >>>> itself > > > > > > > >>>>>> ? > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> If it is just plug-in library and a feature list > > > > > > > >>>>>>>>>> document of > > > > > > > >>>> the > > > > > > > >>>>>>>>>> features in personal repository, then there is no > > > > > > > >>>>>>>>>> need to fork > > > > > > > >>>>>>> Pulsar > > > > > > > > > > [message truncated...]
