Hi - I renamed the PIP PR to include the PIP number - PIP-367. - https://github.com/apache/pulsar/pull/23061
I think that you have covered concerns and it is time to start a separate VOTE thread. Assuming that the PIP passes then we can create the new repository by following your example and adjust the files to best fit policies. Best Regards, Dave > On Jul 29, 2024, at 7:45 PM, xiangying meng <xiangy...@apache.org> wrote: > > Dear Dave and Enrico, > > Thank you for your reply and valuable comments. I think the point that > needs to be clarified now is whether this repository needs to be > released. In the original idea, the functions in this repository are > not stable. It will be used to collect various experimental functions > and extended plug-in implementations. > The risk is mentioned in the first version of the proposal. However, > due to lack of experience, I failed to clarify the importance of > release, so there is no clear release description. Thanks to Dave for > helping to further clarify this matter. > > - For experimental functions and customized functions that require > modifying the Pulsar source code, I think they must not be released. > - For the plug-in implementation of Pulsar extension, it cannot be > promised to users that it is stable, so it cannot be released, too. > > Therefore, we further clarify the positioning of this repository. > > - Collect user needs and their various customized plug-in implementations > - Collect and organize various experimental functions and customized functions > - Collect user best practices > > The admission standard of the warehouse is that this function can work > normally and meet the needs, but it is not necessarily stable, so it > will not be released. > > With the continuous feedback from users, this feature may become > stable, and eventually someone (new contributor or original > contributor) may move this feature to a stable releaseable place. That > is, move it to the `appropriate Pulsar repository and component` you > mentioned. > > In addition, regarding the SECURITY.md file, I have added it to the > repository. [0] > > Best Regards > > xiangying > > [0] - > https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/SECURITY.md > > On Mon, Jul 29, 2024 at 10:50 PM Dave Fisher <w...@apache.org> wrote: >> >> >> >>> On Jul 29, 2024, at 6:37 AM, Enrico Olivelli <eolive...@gmail.com> wrote: >>> >>> Il giorno lun 29 lug 2024 alle ore 11:55 steven lu <lushiji2...@gmail.com> >>> ha scritto: >>> >>>> Different from pulsar-adapters, we added some instructions and examples >>>> here https://github.com/StevenLuMT/pulsar-java-contrib >>> >>> >>> This example is clear, thanks ! >> >> From your README in the repository. >> >>> Pulsar java contrib is to provide a non-core code maintenance repository to >>> collect plugin implementations, personalized features, experimental >>> features, and best practices from users. >> >> These are examples which anyone can use at their own risk and the PMC does >> not intend to make RELEASES. Is this correct? If so then that should be very >> clear. >> >> Anything the community decides to support for the future in a release should >> be moved into the appropriate Pulsar repository and component. >> >> If this were made clear then I would be a +1. >> >> Security vulnerabilities found would still require care and attention from >> the PMC due to the private nature of this reporting. You will need a >> SECURITY.md file. >> >> Best, >> Dave >> >>> >>> Let's put it this way: +0 >>> I am not against this proposal, but I would support it if we see that the >>> Community and in particular the PMC >>> is willing to maintain it. >>> >>> If the community is not willing to maintain it (and we already have >>> problems with pulsar-adapters for instance) then >>> it is only like adding another dead repository plus the responsibility to >>> fix security issues and keep the examples up to date with the best >>> practices as long as Pulsar evolves >>> >>> If I were you I would start a well curated github repository and encourage >>> people to contribute their code snippets to it. >>> >>> That said, I won't VOTE against the proposal >>> >>> Thanks >>> >>> Enrico >>> >>> >>> >>>> >>>> >>>> Enrico Olivelli <eolive...@gmail.com> 于2024年7月23日周二 17:45写道： >>>> >>>>> Probably this repository is already what you want to do: >>>>> https://github.com/apache/pulsar-adapters >>>>> >>>>> It contains a lot of different stuff that we don't keep in the main >>>> pulsar >>>>> repository. >>>>> >>>>> Enrico >>>>> >>>>> >>>>> Il giorno mar 23 lug 2024 alle ore 09:51 Jia Zhai <zhai...@apache.org> >>>> ha >>>>> scritto: >>>>> >>>>>> Thanks for the proposal. It is a good idea. It is also not a bad idea >>>> to >>>>>> incubate it outside of Pulsar official repository at first. >>>>>> >>>>>> On Tue, Jul 23, 2024 at 2:59 PM Aurora Twinkle < >>>> foreverlove...@gmail.com >>>>>> >>>>>> wrote: >>>>>> >>>>>>> Hi: >>>>>>> I think this is a very good idea. >>>>>>> >>>>>>> In the native pulsar warehouse, there are some interfaces that have >>>> no >>>>>>> default implementation classes, such as ConsumerInterceptor, >>>>>>> ProducerInterceptor. Users must implement the interface themselves, >>>>>>> which increases the cost of use. If there is a plug-in library >>>>>>> mentioned in this PIP, we can provide some common interface >>>>>>> implementations in the plug-in library, such as mertics collection >>>>>>> interceptor for producer and consumer. >>>>>>> >>>>>>> Best Regards, >>>>>>> AuroraTwinkle >>>>>>> >>>>>>> xiangying meng <xiangyingme...@gmail.com> 于2024年7月23日周二 11:25写道： >>>>>>>> >>>>>>>> Hello Wen Zhi, >>>>>>>> >>>>>>>> This project is a new repository similar to a connector, and it >>>> will >>>>>>>> not make any modifications to Pulsar itself. [0] >>>>>>>> >>>>>>>> Its code part is simply a plugin library, based on Pulsar's >>>> external >>>>>>>> interfaces, providing a rich set of implementations. It only >>>> retains >>>>> a >>>>>>>> "curated list", which is a series of pointers to personalized >>>>> features >>>>>>>> and experimental features in personal repositories for reference >>>> and >>>>>>>> inspiration. If these experimental features are recognized by other >>>>>>>> users, they can also be gradually merged into the Pulsar main >>>>>>>> repository. However, it will not fork the content of the Pulsar >>>>>>>> repository for modification in the new repository. >>>>>>>> This is the improvement we made after discussion in the last >>>> email. I >>>>>>>> will synchronize it to the proposal later. >>>>>>>> Finally, thank you for your valuable suggestions, and we look >>>> forward >>>>>>>> to your continued participation. >>>>>>>> >>>>>>>> [0] - https://github.com/apache/pulsar-connectors >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Xiangying >>>>>>>> >>>>>>>> On Tue, Jul 23, 2024 at 10:36 AM WenZhi Feng < >>>> thetumb...@apache.org> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, xiangying, >>>>>>>>> First of all, thanks for driving the proposal. >>>>>>>>> I have some questions. >>>>>>>>> 1. Will this repository maintained by committers/pmc? If the >>>>> answer >>>>>>> is yes, there will be too many new modules introduced into pulsar and >>>>>> hard >>>>>>> to maintain. If the answer is no, i think we have better not to >>>> include >>>>>> it >>>>>>> into pulsar official repository. >>>>>>>>> 2. For those plugin-in feature, we may create a new repository >>>>> like >>>>>>> various connectors for pulsar. But for those core code change on >>>>> pulsar, >>>>>>> how can we seperate it with pulsar repository? >>>>>>>>> It is thrilling that users contribute various new features to >>>>>> apache >>>>>>> community, enriching the ecosystem. I think we should find out a good >>>>> way >>>>>>> to realize it. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Wenzhi Feng. >>>>>>>>> >>>>>>>>> On 2024/07/22 17:40:24 xiangying meng wrote: >>>>>>>>>> Thank you for your feedback, let's further clarify this matter. >>>>>>>>>> >>>>>>>>>> The original design of this proposal is a new contributor >>>>>> repository, >>>>>>>>>> and provides two ways to contribute. >>>>>>>>>> The first one: >>>>>>>>>> Contributors contribute through PR and accept code review, and >>>>> can >>>>>>>>>> only be merged into the main branch after reviewed. The code >>>>> merged >>>>>>> in >>>>>>>>>> this way is various plug-ins based on Pulsar's external >>>>> interface. >>>>>> It >>>>>>>>>> does not contain any pulsar code, only Pulsar's dependencies. >>>>>>>>>> The second one: >>>>>>>>>> Features that require modifications to the Pulsar main >>>> repository >>>>>>> code >>>>>>>>>> and may conflict with later versions. My initial idea was for >>>>>>>>>> contributors to create branches and then write the commit id >>>> and >>>>>>>>>> features description into the document after the PR is >>>> reviewed. >>>>>>>>>> >>>>>>>>>> However, after the discussion just now, I agree that the second >>>>> one >>>>>>>>>> may not be maintained in the branch of the new repository, but >>>> in >>>>>> the >>>>>>>>>> contributor's personal repository. >>>>>>>>>> >>>>>>>>>> Based on the above content, let's continue to look at the >>>>> question >>>>>>> you raised. >>>>>>>>>> >>>>>>>>>>> - How can the Pulsar community maintain a fork of Pulsar >>>> itself >>>>>> ? >>>>>>>>>> >>>>>>>>>> If it is just plug-in library and a feature list document of >>>> the >>>>>>>>>> features in personal repository, then there is no need to fork >>>>>>> Pulsar >>>>>>>>>> itself. >>>>>>>>>> >>>>>>>>>>> - Do you want to "cut releases" out of this fork ? Who is >>>> going >>>>>> to >>>>>>> validate them ? Who is responsible for security bugs? >>>>>>>>>> >>>>>>>>>> We will only need to maintain a documents of feature and >>>> plugins >>>>>> list >>>>>>>>>> that have been reviewed by PR. We will not make changes to the >>>>>> Pulsar >>>>>>>>>> code itself. There is no fork of Pulsar. It will have its own >>>>>> version >>>>>>>>>> release process, just like Pulsar. >>>>>>>>>> >>>>>>>>>> -> Who is going to use this code ? >>>>>>>>>> >>>>>>>>>> IMO, providing various implementation plugins based on Pulsar's >>>>>>>>>> external interface is meaningful in itself. While introducing >>>>>>> Pulsar's >>>>>>>>>> own dependencies, introducing plugin libraries to use various >>>>>> already >>>>>>>>>> implemented plugins can reduce development costs. I think some >>>>>> people >>>>>>>>>> will be happy to use them. >>>>>>>>>> >>>>>>>>>> As for the other functions in the document, maintained in >>>>> personal >>>>>>>>>> repositories, they can be used as a reference and provided to >>>>>>>>>> companies that are capable of solving security issues and bugs. >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Xiangying >>>>>>>>>> >>>>>>>>>> On Tue, Jul 23, 2024 at 12:03 AM Enrico Olivelli < >>>>>>> eolive...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Il giorno lun 22 lug 2024 alle ore 17:34 xiangying meng < >>>>>>>>>>> xiangyingme...@gmail.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>>>> thanks for your proposal, we need more and more energy in >>>>> the >>>>>>> project. >>>>>>>>>>>> >>>>>>>>>>>> Yes, it requires a lot of participation, but sometimes we >>>>> also >>>>>>> need a >>>>>>>>>>>> start. >>>>>>>>>>>> >>>>>>>>>>>>> Also we allow only "committers" to commit patches to the >>>>>>> repositories >>>>>>>>>>>> that are under the responsibility of the Apache Pulsar >>>> PMC, I >>>>>>> don't know >>>>>>>>>>>> how we can make this work with a "official fork" >>>>>>>>>>>> >>>>>>>>>>>>> An alternative is to have a "curated list" of links to >>>>>> personal >>>>>>> projects, >>>>>>>>>>>> but we keep the responsibility over the code on the code >>>>> owners >>>>>>> and >>>>>>>>>>>> not on the Pulsar PMC >>>>>>>>>>>> >>>>>>>>>>>> Yes, this is a "curated list" link, but it points to the >>>>> branch >>>>>>> and >>>>>>>>>>>> commit ID created by the contributor himself. In addition, >>>>> for >>>>>>> some >>>>>>>>>>>> interface implementation classes, they can be placed in the >>>>>> main >>>>>>>>>>>> branch of the new contributor's repository and then added >>>> to >>>>>> the >>>>>>>>>>>> "curated list". Only committors or maintainers can add >>>>> content >>>>>>> to the >>>>>>>>>>>> "curated list". I have detailed these in the proposal. >>>>>>>>>>>> >>>>>>>>>>>> But no matter which method, each function will be marked in >>>>> the >>>>>>>>>>>> document, that is, in the "Featured List" who contributed >>>>> it. I >>>>>>>>>>>> understand that this feature is more of a reference >>>> feature. >>>>>>> Allow >>>>>>>>>>>> users to choose these customized and risky features to >>>>>> customize >>>>>>> their >>>>>>>>>>>> own versions. This is very useful for some large companies >>>>> with >>>>>>>>>>>> development capabilities. At the same time, its >>>>> responsibility >>>>>>> should >>>>>>>>>>>> belong to the user, although we will also point out who is >>>>> the >>>>>>>>>>>> contributor of each feature he chooses. But when using >>>>>>> innovative or >>>>>>>>>>>> customized features, users should also be clear about the >>>>>> risks. >>>>>>> I >>>>>>>>>>>> also wrote about the risks of contributor repositories in >>>> the >>>>>>>>>>>> proposal. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The main point is that we cannot grant "write" permissions to >>>>>>> repositories >>>>>>>>>>> owned by the Pulsar project (the PMC) >>>>>>>>>>> to people who are not "committers". This is a legal thing, no >>>>>>> exceptions. >>>>>>>>>>> >>>>>>>>>>> So any contribution must be in the form of a PR, like for any >>>>>>> other patches >>>>>>>>>>> contributes to one of the github repositories owned by the >>>>> Pulsar >>>>>>> project >>>>>>>>>>> >>>>>>>>>>> A couple of questions: >>>>>>>>>>> - How can the Pulsar community maintain a fork of Pulsar >>>>>> itself ? >>>>>>>>>>> - Do you want to "cut releases" out of this fork ? Who is >>>> going >>>>>> to >>>>>>> validate >>>>>>>>>>> them ? Who is responsible for security bugs ? >>>>>>>>>>> - Who is going to use this code ? >>>>>>>>>>> >>>>>>>>>>> I think that when you want to contribute a feature, you can >>>> do >>>>> it >>>>>>> with a PR >>>>>>>>>>> or with the PIP process (if it is bigger), and it is fine to >>>>> mark >>>>>>> it as >>>>>>>>>>> "experimental". >>>>>>>>>>> >>>>>>>>>>> Then the community decides whether to support it or not. >>>>>>>>>>> >>>>>>>>>>> When a patch is committed to our repository then it is the >>>>>>> responsibility >>>>>>>>>>> of all of us (the PMC and the committers) to maintain it >>>> almost >>>>>>> forever, >>>>>>>>>>> unless we decide to delete it. >>>>>>>>>>> >>>>>>>>>>> This is a legal thing, we cannot accumulate code without >>>> being >>>>>>> responsible >>>>>>>>>>> for that. >>>>>>>>>>> >>>>>>>>>>> I am happy to clarify more if needed, and I hope that other >>>> in >>>>>> the >>>>>>>>>>> community can help in understanding if this is doable of not >>>>>>>>>>> >>>>>>>>>>> Enrico >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> In addition, thank you for your valuable feedback. >>>>>>>>>>>> >>>>>>>>>>>> Best regards >>>>>>>>>>>> Xiangying >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 22, 2024 at 10:55 PM Enrico Olivelli < >>>>>>> eolive...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> thanks for your proposal, we need more and more energy in >>>>> the >>>>>>> project. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think that this is a good idea. >>>>>>>>>>>>> >>>>>>>>>>>>> The main reason is that as an ASF project we must >>>> guarantee >>>>>>> the quality >>>>>>>>>>>> and >>>>>>>>>>>>> especially the security of what we provide to the public. >>>>>>>>>>>>> >>>>>>>>>>>>> Such kinds of repositories tend to become full of stale >>>>> code >>>>>>> that can >>>>>>>>>>>>> become a source of security issues. >>>>>>>>>>>>> >>>>>>>>>>>>> Also we allow only "committers" to commit patches to the >>>>>>> repositories >>>>>>>>>>>> that >>>>>>>>>>>>> are under the responsibility of the Apache Pulsar PMC, I >>>>>> don't >>>>>>> know how >>>>>>>>>>>> we >>>>>>>>>>>>> can make this work with a "official fork" >>>>>>>>>>>>> >>>>>>>>>>>>> Maybe I misunderstood the goal ? >>>>>>>>>>>>> >>>>>>>>>>>>> An alternative is to have a "curated list" of links to >>>>>>> personal projects, >>>>>>>>>>>>> but we keep the responsibility over the code on the code >>>>>>> owners and not >>>>>>>>>>>> on >>>>>>>>>>>>> the Pulsar PMC >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards >>>>>>>>>>>>> Enrico Olivelli >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno lun 22 lug 2024 alle ore 15:15 steven lu < >>>>>>>>>>>> lushiji2...@gmail.com> >>>>>>>>>>>>> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>>> We think this is a very good solution, >>>>>>>>>>>>>> Our XHS team(@liangyepianzhou < >>>>>>> https://github.com/liangyepianzhou> >>>>>>>>>>>>>> @AuroraTwinkle <https://github.com/AuroraTwinkle> >>>>>>> @StevenLuMT >>>>>>>>>>>>>> <https://github.com/StevenLuMT> @cai152 < >>>>>>> https://github.com/cai152>) >>>>>>>>>>>> will >>>>>>>>>>>>>> submit a few to pulsar-java-contrib >>>>>>>>>>>>>> <https://github.com/StevenLuMT/pulsar-java-contrib>( >>>>>>>>>>>>>> https://github.com/StevenLuMT/pulsar-java-contrib) as >>>> a >>>>>>> primer to let >>>>>>>>>>>>>> everyone know the role of this library >>>>>>>>>>>>>> >>>>>>>>>>>>>> xiangying meng <xiangyingme...@gmail.com> >>>> 于2024年7月22日周一 >>>>>>> 16:33写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello Pulsar Community, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I hope this message finds you well. I'm reaching out >>>> to >>>>>>> propose the >>>>>>>>>>>>>>> establishment of a Pulsar Contributor Repository. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This new initiative is designed to provide a >>>> dedicated >>>>>>> space for >>>>>>>>>>>>>>> experimental and community-driven features that >>>>>> complement >>>>>>> our core >>>>>>>>>>>>>>> offerings. This repository would serve as a space >>>> for >>>>>>> non-core >>>>>>>>>>>>>>> features and customizations, allowing for greater >>>>>>> flexibility and >>>>>>>>>>>>>>> community involvement. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The proposal can be viewed at this GitHub link: >>>>>>>>>>>>>>> https://github.com/apache/pulsar/pull/23061/files >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm looking forward to your thoughts and feedback. >>>>> Please >>>>>>> feel free >>>>>>>>>>>> to >>>>>>>>>>>>>>> share them on the Pulsar Developer Mailing List. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you in advance for your time and consideration. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Xiangying >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>
