Hi Yunze, A few of your points are of concern and to me could be good arguments for this extra code repository.
On 2024/08/05 13:25:24 Yunze Xu wrote: > Let's ignore the last sentence I mentioned. I mentioned my suspicion > because I have concerns about the lifetime of this repo. It's > clarified privately so I'm apologizing for my conjecture about the > company. So let's focus on the topic itself. Thank you what our employers are doing with or without Pulsar must be of no concern to the merits of each of us who are here as INDIVIDUALS. > > > > I just thought again about the "best practice" part. Why not > > contribute it to the Apache official website > > (https://github.com/apache/pulsar-site)? One way to use this repository is to develop code examples and best practices in a shared environment in public. > > > > > > TL;DR, such a repository is not necessary to be contributed to Apache. > > > > > > Actually, only the "Collect user best practices" point makes sense to > > > me. The reason to have so many pluggables is exactly to avoid the core > > > repo being not so bloated. > > > > > > Yeah, OTel is a good example. But IMHO, Pulsar is far less populated > > > than OTel. In the current phase, it's more important to improve the > > > code quality of such plugins, including > > > - Metadata store. From my personal perspective, I don't suggest making > > > InMemory, Etcd, Oxia as the built-in implementations. Having ZK as > > > default and RocksDB for a lightweight standalone is enough. True, but these other metadata stores are being used and developing examples in public can only help others understand how to make use of the many years effort to enable these other metadata stores without making these fully builtin. Etcd is a project that has very wide use and it may be good for Pulsar to share how to use it. Apache JClouds is going to the Apache Attic soon. Pulsar depends on it. Perhaps this repository could allow development of an alternative approach in a pre-PIP don’t break the CI manner. > > > - Load Manager. If you have the demand to implement a customized load > > > manager (like me), you will understand it. The extensible load manager > > > introduced from PIP-192 makes things better. But it's still a mess. > > > See how many `isLoadManagerExtensionEnabled` calls in the repo. > > > - Schema Registry. It's not pluggable yet. But I've heard many users > > > want a more powerful implementation. > > > > > > These three typical plugins above are really not pluggable enough. > > > Before adding them to such a repository, it's more worth taking some > > > time to revisit the implementation and make them better. Just like the > > > producer and consumer interceptors, if some implementations are > > > general enough, we should accept them in the core repo or create > > > another repo for them. > > > > > > Besides, these pluggables provide a way for companies to implement its > > > own private plugins as the sell points. For example, I really don't > > > think my employer will be glad to contribute ideas of our internal > > > plugins to this repo. Your ability to contribute is defined by a combination of the ICLA that you signed and your work contract with your employer. If you cannot contribute then please do not. If others have no such restriction then it is to the benefit of the Pulsar community for them to be enabled. > > > > > > So. It's -1 for me. But I think it's good to incubate this repo for > > > some time and see if there are many people willing to contribute, > > > rather than the XHS team itself. I do not see ‘XHS’, I see an individual who has a good idea about how to become more involved in contributing to Pulsar in a more incremental manner. > > > > > > BTW, I'm also curious about the position of Pulsar in XHS since I see > > > your company is adopting AutoMQ recently. I'm suspicious that > > > contributing such a repo into Apache is a KPI in the internal > > > competition. Best, Dave > > > > > > Thanks, > > > Yunze > > > > > > On Sat, Aug 3, 2024 at 5:37 PM xiangying meng <xiangy...@apache.org> > > > wrote: > > > > > > > > Dear all, > > > > > > > > I would like to express my appreciation to all of you who have shown > > > > support for my recent proposal. Your feedback and encouragement are > > > > greatly appreciated and have been instrumental in moving the > > > > discussion forward. > > > > > > > > I am pleased to announce that I have initiated a vote on this > > > > proposal. You can find the details and participate in the voting > > > > through the following link: > > > > > > > > https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y > > > > > > > > Your active participation is crucial for the success of this > > > > initiative, and I look forward to a positive outcome. > > > > > > > > Once again, thank you for your valuable input and support. > > > > > > > > Best regards, > > > > Xiangying > > > > > > > > On Sat, Aug 3, 2024 at 3:38 AM Dave Fisher <w...@apache.org> wrote: > > > > > > > > > > Hi - I renamed the PIP PR to include the PIP number - PIP-367. - > > > > > https://github.com/apache/pulsar/pull/23061 > > > > > > > > > > I think that you have covered concerns and it is time to start a > > > > > separate VOTE thread. > > > > > > > > > > Assuming that the PIP passes then we can create the new repository by > > > > > following your example and adjust the files to best fit policies. > > > > > > > > > > Best Regards, > > > > > Dave > > > > > > > > > > > On Jul 29, 2024, at 7:45 PM, xiangying meng <xiangy...@apache.org> > > > > > > wrote: > > > > > > > > > > > > Dear Dave and Enrico, > > > > > > > > > > > > Thank you for your reply and valuable comments. I think the point > > > > > > that > > > > > > needs to be clarified now is whether this repository needs to be > > > > > > released. In the original idea, the functions in this repository are > > > > > > not stable. It will be used to collect various experimental > > > > > > functions > > > > > > and extended plug-in implementations. > > > > > > The risk is mentioned in the first version of the proposal. However, > > > > > > due to lack of experience, I failed to clarify the importance of > > > > > > release, so there is no clear release description. Thanks to Dave > > > > > > for > > > > > > helping to further clarify this matter. > > > > > > > > > > > > - For experimental functions and customized functions that require > > > > > > modifying the Pulsar source code, I think they must not be released. > > > > > > - For the plug-in implementation of Pulsar extension, it cannot be > > > > > > promised to users that it is stable, so it cannot be released, too. > > > > > > > > > > > > Therefore, we further clarify the positioning of this repository. > > > > > > > > > > > > - Collect user needs and their various customized plug-in > > > > > > implementations > > > > > > - Collect and organize various experimental functions and > > > > > > customized functions > > > > > > - Collect user best practices > > > > > > > > > > > > The admission standard of the warehouse is that this function can > > > > > > work > > > > > > normally and meet the needs, but it is not necessarily stable, so it > > > > > > will not be released. > > > > > > > > > > > > With the continuous feedback from users, this feature may become > > > > > > stable, and eventually someone (new contributor or original > > > > > > contributor) may move this feature to a stable releaseable place. > > > > > > That > > > > > > is, move it to the `appropriate Pulsar repository and component` you > > > > > > mentioned. > > > > > > > > > > > > In addition, regarding the SECURITY.md file, I have added it to the > > > > > > repository. [0] > > > > > > > > > > > > Best Regards > > > > > > > > > > > > xiangying > > > > > > > > > > > > [0] - > > > > > > https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/SECURITY.md > > > > > > > > > > > > On Mon, Jul 29, 2024 at 10:50 PM Dave Fisher <w...@apache.org> > > > > > > wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >>> On Jul 29, 2024, at 6:37 AM, Enrico Olivelli > > > > > >>> <eolive...@gmail.com> wrote: > > > > > >>> > > > > > >>> Il giorno lun 29 lug 2024 alle ore 11:55 steven lu > > > > > >>> <lushiji2...@gmail.com> > > > > > >>> ha scritto: > > > > > >>> > > > > > >>>> Different from pulsar-adapters, we added some instructions and > > > > > >>>> examples > > > > > >>>> here https://github.com/StevenLuMT/pulsar-java-contrib > > > > > >>> > > > > > >>> > > > > > >>> This example is clear, thanks ! > > > > > >> > > > > > >> From your README in the repository. > > > > > >> > > > > > >>> Pulsar java contrib is to provide a non-core code maintenance > > > > > >>> repository to collect plugin implementations, personalized > > > > > >>> features, experimental features, and best practices from users. > > > > > >> > > > > > >> These are examples which anyone can use at their own risk and the > > > > > >> PMC does not intend to make RELEASES. Is this correct? If so then > > > > > >> that should be very clear. > > > > > >> > > > > > >> Anything the community decides to support for the future in a > > > > > >> release should be moved into the appropriate Pulsar repository and > > > > > >> component. > > > > > >> > > > > > >> If this were made clear then I would be a +1. > > > > > >> > > > > > >> Security vulnerabilities found would still require care and > > > > > >> attention from the PMC due to the private nature of this > > > > > >> reporting. You will need a SECURITY.md file. > > > > > >> > > > > > >> Best, > > > > > >> Dave > > > > > >> > > > > > >>> > > > > > >>> Let's put it this way: +0 > > > > > >>> I am not against this proposal, but I would support it if we see > > > > > >>> that the > > > > > >>> Community and in particular the PMC > > > > > >>> is willing to maintain it. > > > > > >>> > > > > > >>> If the community is not willing to maintain it (and we already > > > > > >>> have > > > > > >>> problems with pulsar-adapters for instance) then > > > > > >>> it is only like adding another dead repository plus the > > > > > >>> responsibility to > > > > > >>> fix security issues and keep the examples up to date with the best > > > > > >>> practices as long as Pulsar evolves > > > > > >>> > > > > > >>> If I were you I would start a well curated github repository and > > > > > >>> encourage > > > > > >>> people to contribute their code snippets to it. > > > > > >>> > > > > > >>> That said, I won't VOTE against the proposal > > > > > >>> > > > > > >>> Thanks > > > > > >>> > > > > > >>> Enrico > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>>> > > > > > >>>> > > > > > >>>> Enrico Olivelli <eolive...@gmail.com> 于2024年7月23日周二 17:45写道： > > > > > >>>> > > > > > >>>>> Probably this repository is already what you want to do: > > > > > >>>>> https://github.com/apache/pulsar-adapters > > > > > >>>>> > > > > > >>>>> It contains a lot of different stuff that we don't keep in the > > > > > >>>>> main > > > > > >>>> pulsar > > > > > >>>>> repository. > > > > > >>>>> > > > > > >>>>> Enrico > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> Il giorno mar 23 lug 2024 alle ore 09:51 Jia Zhai > > > > > >>>>> <zhai...@apache.org> > > > > > >>>> ha > > > > > >>>>> scritto: > > > > > >>>>> > > > > > >>>>>> Thanks for the proposal. It is a good idea. It is also not a > > > > > >>>>>> bad idea > > > > > >>>> to > > > > > >>>>>> incubate it outside of Pulsar official repository at first. > > > > > >>>>>> > > > > > >>>>>> On Tue, Jul 23, 2024 at 2:59 PM Aurora Twinkle < > > > > > >>>> foreverlove...@gmail.com > > > > > >>>>>> > > > > > >>>>>> wrote: > > > > > >>>>>> > > > > > >>>>>>> Hi: > > > > > >>>>>>> I think this is a very good idea. > > > > > >>>>>>> > > > > > >>>>>>> In the native pulsar warehouse, there are some interfaces > > > > > >>>>>>> that have > > > > > >>>> no > > > > > >>>>>>> default implementation classes, such as ConsumerInterceptor, > > > > > >>>>>>> ProducerInterceptor. Users must implement the interface > > > > > >>>>>>> themselves, > > > > > >>>>>>> which increases the cost of use. If there is a plug-in library > > > > > >>>>>>> mentioned in this PIP, we can provide some common interface > > > > > >>>>>>> implementations in the plug-in library, such as mertics > > > > > >>>>>>> collection > > > > > >>>>>>> interceptor for producer and consumer. > > > > > >>>>>>> > > > > > >>>>>>> Best Regards, > > > > > >>>>>>> AuroraTwinkle > > > > > >>>>>>> > > > > > >>>>>>> xiangying meng <xiangyingme...@gmail.com> 于2024年7月23日周二 > > > > > >>>>>>> 11:25写道： > > > > > >>>>>>>> > > > > > >>>>>>>> Hello Wen Zhi, > > > > > >>>>>>>> > > > > > >>>>>>>> This project is a new repository similar to a connector, and > > > > > >>>>>>>> it > > > > > >>>> will > > > > > >>>>>>>> not make any modifications to Pulsar itself. [0] > > > > > >>>>>>>> > > > > > >>>>>>>> Its code part is simply a plugin library, based on Pulsar's > > > > > >>>> external > > > > > >>>>>>>> interfaces, providing a rich set of implementations. It only > > > > > >>>> retains > > > > > >>>>> a > > > > > >>>>>>>> "curated list", which is a series of pointers to personalized > > > > > >>>>> features > > > > > >>>>>>>> and experimental features in personal repositories for > > > > > >>>>>>>> reference > > > > > >>>> and > > > > > >>>>>>>> inspiration. If these experimental features are recognized > > > > > >>>>>>>> by other > > > > > >>>>>>>> users, they can also be gradually merged into the Pulsar main > > > > > >>>>>>>> repository. However, it will not fork the content of the > > > > > >>>>>>>> Pulsar > > > > > >>>>>>>> repository for modification in the new repository. > > > > > >>>>>>>> This is the improvement we made after discussion in the last > > > > > >>>> email. I > > > > > >>>>>>>> will synchronize it to the proposal later. > > > > > >>>>>>>> Finally, thank you for your valuable suggestions, and we look > > > > > >>>> forward > > > > > >>>>>>>> to your continued participation. > > > > > >>>>>>>> > > > > > >>>>>>>> [0] - https://github.com/apache/pulsar-connectors > > > > > >>>>>>>> > > > > > >>>>>>>> Best Regards, > > > > > >>>>>>>> Xiangying > > > > > >>>>>>>> > > > > > >>>>>>>> On Tue, Jul 23, 2024 at 10:36 AM WenZhi Feng < > > > > > >>>> thetumb...@apache.org> > > > > > >>>>>>> wrote: > > > > > >>>>>>>>> > > > > > >>>>>>>>> Hi, xiangying, > > > > > >>>>>>>>> First of all, thanks for driving the proposal. > > > > > >>>>>>>>> I have some questions. > > > > > >>>>>>>>> 1. Will this repository maintained by committers/pmc? If the > > > > > >>>>> answer > > > > > >>>>>>> is yes, there will be too many new modules introduced into > > > > > >>>>>>> pulsar and > > > > > >>>>>> hard > > > > > >>>>>>> to maintain. If the answer is no, i think we have better not > > > > > >>>>>>> to > > > > > >>>> include > > > > > >>>>>> it > > > > > >>>>>>> into pulsar official repository. > > > > > >>>>>>>>> 2. For those plugin-in feature, we may create a new > > > > > >>>>>>>>> repository > > > > > >>>>> like > > > > > >>>>>>> various connectors for pulsar. But for those core code change > > > > > >>>>>>> on > > > > > >>>>> pulsar, > > > > > >>>>>>> how can we seperate it with pulsar repository? > > > > > >>>>>>>>> It is thrilling that users contribute various new features > > > > > >>>>>>>>> to > > > > > >>>>>> apache > > > > > >>>>>>> community, enriching the ecosystem. I think we should find > > > > > >>>>>>> out a good > > > > > >>>>> way > > > > > >>>>>>> to realize it. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks, > > > > > >>>>>>>>> Wenzhi Feng. > > > > > >>>>>>>>> > > > > > >>>>>>>>> On 2024/07/22 17:40:24 xiangying meng wrote: > > > > > >>>>>>>>>> Thank you for your feedback, let's further clarify this > > > > > >>>>>>>>>> matter. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> The original design of this proposal is a new contributor > > > > > >>>>>> repository, > > > > > >>>>>>>>>> and provides two ways to contribute. > > > > > >>>>>>>>>> The first one: > > > > > >>>>>>>>>> Contributors contribute through PR and accept code review, > > > > > >>>>>>>>>> and > > > > > >>>>> can > > > > > >>>>>>>>>> only be merged into the main branch after reviewed. The > > > > > >>>>>>>>>> code > > > > > >>>>> merged > > > > > >>>>>>> in > > > > > >>>>>>>>>> this way is various plug-ins based on Pulsar's external > > > > > >>>>> interface. > > > > > >>>>>> It > > > > > >>>>>>>>>> does not contain any pulsar code, only Pulsar's > > > > > >>>>>>>>>> dependencies. > > > > > >>>>>>>>>> The second one: > > > > > >>>>>>>>>> Features that require modifications to the Pulsar main > > > > > >>>> repository > > > > > >>>>>>> code > > > > > >>>>>>>>>> and may conflict with later versions. My initial idea was > > > > > >>>>>>>>>> for > > > > > >>>>>>>>>> contributors to create branches and then write the commit > > > > > >>>>>>>>>> id > > > > > >>>> and > > > > > >>>>>>>>>> features description into the document after the PR is > > > > > >>>> reviewed. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> However, after the discussion just now, I agree that the > > > > > >>>>>>>>>> second > > > > > >>>>> one > > > > > >>>>>>>>>> may not be maintained in the branch of the new repository, > > > > > >>>>>>>>>> but > > > > > >>>> in > > > > > >>>>>> the > > > > > >>>>>>>>>> contributor's personal repository. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Based on the above content, let's continue to look at the > > > > > >>>>> question > > > > > >>>>>>> you raised. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> - How can the Pulsar community maintain a fork of Pulsar > > > > > >>>> itself > > > > > >>>>>> ? > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> If it is just plug-in library and a feature list document > > > > > >>>>>>>>>> of > > > > > >>>> the > > > > > >>>>>>>>>> features in personal repository, then there is no need to > > > > > >>>>>>>>>> fork > > > > > >>>>>>> Pulsar > > > > > > [message truncated...]
