Cool to see that the Bahir people are so open and helpful! Concerning moving the connectors: I think we should set up a vote thread, or at least a discussion with the possibility to object. Removing someones code from the repository is a bit of a sensitive thing in my experience, so let's make sure everybody is behind that decision before we proceed.
On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger <rmetz...@apache.org> wrote: > Bahir is now creating a second repository "bahir-flink" for our connectors: > https://issues.apache.org/jira/browse/INFRA-12440 > > If there are no objections here on the dev list, I would like to move the > "flume" and "redis" streaming connector to Bahir. > Once they are in, and the file structure at Bahir exists, I'll update our > contribution guidelines to point people to Bahir. > > On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger <rmetz...@apache.org> > wrote: > > > Hi, > > > > I just wanted to let you know that I've started a discussion at the Bahir > > dev list [1]. They seem to be very open to give some of our streaming > > connectors a new home. > > We are currently discussing some specifics there (whether to put the code > > into the same repository or into separate ones). Also, they asked for > some > > help by Flink committers / contributors to help with the reviewing of > > incoming contributions. > > I'm definitively willing to help out to give the community there a good > > start. > > > > > > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html > > > > > > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels <m...@apache.org> > > wrote: > > > >> Hi Robert, > >> > >> We had this discussion before when I suggested to use an external > >> repository to manage connectors. Ever since I have come to the > >> conclusion that the overhead of maintaining two source repositories > >> along with maintaining code and integration, documentation, and CI, is > >> not worth the effort. Thanks for bringing up the discussion again. > >> > >> I think the most important issues with moving connectors out of the > >> core repository are > >> > >> 1) Maintenance > >> Connectors need to be maintained. Otherwise, they are worthless. > >> > >> 2) Plugability > >> Connectors need to be easy to use for the user. Otherwise, nobody will > >> use them. > >> > >> What is the difference between #1, #3 and #4? I don't see a difference > >> except that #3 and #4 are central repositories. Still, maintenance and > >> plugability are very unclear. > >> > >> Only #2 is left :) Apache Bahir looks like a promising project. Let's > >> see how the community reacts. > >> > >> Cheers, > >> Max > >> > >> > >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetz...@apache.org> > >> wrote: > >> > Thank you for your responses. I will get in touch with the Bahir > >> community > >> > to see what they are thinking about this. > >> > Once we know a bit more about the details of such a collaboration, we > >> can > >> > make a final decision here. > >> > > >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrm...@apache.org> > >> wrote: > >> > > >> >> I agree with Stephan that the main problem is maintenance overhead > for > >> the > >> >> Flink community. If we could maintain all connectors ourselves then > >> there > >> >> would not be an immediate need to out source the connectors. Thus, > the > >> >> solution should reduce the workload for the core project. > >> >> > >> >> Personally, I would favour option #2 if Apache Bahir is willing to > add > >> >> Flink as another supported streaming platform. This would give the > >> >> streaming connectors a more prominent home than the personal > >> repository of > >> >> a contributor. > >> >> > >> >> Furthermore, contributing the connectors to Apache Bahir entails that > >> the > >> >> connector artifacts are deployed to a central repository (I assume). > >> That > >> >> way one can more easily add them to one's project instead of having > to > >> >> build and install the code yourself. > >> >> > >> >> I'm also not sure what happens to a public github repository which > has > >> not > >> >> been forked and is then deleted. I would assume that the content is > >> then > >> >> lost. > >> >> > >> >> To create a "flink-connectors" repository independent of Flink sounds > >> quite > >> >> similar to creating another Apache Bahir. I think it would be better > to > >> >> join forces with the existing Bahir community if possible. > >> >> > >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <se...@apache.org> > wrote: > >> >> > >> >> > Thanks Robert for bringing this up. > >> >> > > >> >> > I think that "flink-contrib" will not really solve the problem, > >> because > >> >> the > >> >> > connectors would still have to be maintained by the core community, > >> we > >> >> > would need to guarantee test stability. It will be to a large > extend > >> >> > similar to adding them to "flink-streaming-connectors", except that > >> we > >> >> may > >> >> > declare via the artifact name that the quality is not guaranteed to > >> be > >> >> too > >> >> > high. > >> >> > > >> >> > Regarding moving the code to a separate repository: I think that > also > >> >> only > >> >> > solves the problem, if that repository is organizationally > different > >> from > >> >> > the core clink repository. Different release cycles, different > >> >> maintainers. > >> >> > > >> >> > > >> >> > The quintessence is for me that the connectors need to be handled > by > >> a > >> >> > different organization than the core flink community. > >> >> > That would leave us with > >> >> > - The contributors' own repository > >> >> > - Apache Bahir > >> >> > - An organizationally different "flink-connectors" repository, > >> possibly > >> >> > with different committers / PMC > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai < > >> tzuli...@gmail.com> > >> >> > wrote: > >> >> > > >> >> > > Good point. Discussion for each new connector is also an overhead > >> we > >> >> > should > >> >> > > avoid. > >> >> > > > >> >> > > IMHO, option #2 doesn’t seem like a reasonable staging area. Say > we > >> >> > decide > >> >> > > we’d like to move a connector from Bahir to Flink in the future, > >> >> there’ll > >> >> > > be two of the connector in separate Apache projects. Personally I > >> think > >> >> > > option #2 is more like a way to go if some day we’d like to > migrate > >> >> some > >> >> > of > >> >> > > our currently supported connectors in Flink to Bahir (perhaps an > >> >> > > alternative to the discussion in "Externalizing the Flink > >> connectors”). > >> >> > > > >> >> > > Defining option #1 as a staging area seems nice; the contributor > >> >> notifies > >> >> > > the Flink community of the work by adding it to the 3rd party > >> packages > >> >> > > page, and in the future we can contact the contributor to ask > >> whether > >> >> > > they’d like to migrate the connector to Flink when we see more > user > >> >> > demand > >> >> > > and committer support. > >> >> > > > >> >> > > With this approach, do you think we should assume that future new > >> >> > > connectors always go to option #1 first? If not, I would assume > >> that > >> >> in > >> >> > > the future, Flink JIRAs for new connectors are only opened if > we’d > >> like > >> >> > to > >> >> > > add them directly to Flink (perhaps the contribution guideline > can > >> link > >> >> > to > >> >> > > Flink development Roadmaps like [1] as a reference to connectors > >> that > >> >> the > >> >> > > community has already considered to support?). > >> >> > > > >> >> > > [1] > >> >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm- > >> >> > > 778DAD7eqw5GANwE/edit# > >> >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm- > >> >> > > 778DAD7eqw5GANwE/edit> > >> >> > > > >> >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger ( > >> rmetz...@apache.org) > >> >> > > wrote: > >> >> > > > >> >> > > Hi Gordon, > >> >> > > thank you for your response. > >> >> > > > >> >> > > I agree with your observation that some "staging" area is helpful > >> to > >> >> test > >> >> > > how many contributors / users are interested in a connector. But > I > >> >> wonder > >> >> > > if #1 or #2 can also serve as a staging area: As soon as we see > >> that > >> >> > there > >> >> > > is a lot of interest in a connector, we add it to the main > >> >> > > "flink-*-connectors" directory. > >> >> > > Having too many ways to contribute connectors might make the > >> >> discussions > >> >> > > for each new connector quite complicated. > >> >> > > > >> >> > > You are right, the intention of the "Externalizing the Flink > >> >> connectors” > >> >> > > discussion was more about the release intervals, test time etc. > >> >> > > > >> >> > > Regards, > >> >> > > Robert > >> >> > > > >> >> > > >> >> > >> > > > > >
