Okay, I agree that this is something we should make very clear. I'll start a DISCUSS thread.
On Wed, Aug 17, 2016 at 10:07 AM, Stephan Ewen <se...@apache.org> wrote: > Cool to see that the Bahir people are so open and helpful! > > Concerning moving the connectors: I think we should set up a vote thread, > or at least a discussion with the possibility to object. > Removing someones code from the repository is a bit of a sensitive thing in > my experience, so let's make sure everybody is behind that decision before > we proceed. > > On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger <rmetz...@apache.org> > wrote: > > > Bahir is now creating a second repository "bahir-flink" for our > connectors: > > https://issues.apache.org/jira/browse/INFRA-12440 > > > > If there are no objections here on the dev list, I would like to move the > > "flume" and "redis" streaming connector to Bahir. > > Once they are in, and the file structure at Bahir exists, I'll update our > > contribution guidelines to point people to Bahir. > > > > On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger <rmetz...@apache.org> > > wrote: > > > > > Hi, > > > > > > I just wanted to let you know that I've started a discussion at the > Bahir > > > dev list [1]. They seem to be very open to give some of our streaming > > > connectors a new home. > > > We are currently discussing some specifics there (whether to put the > code > > > into the same repository or into separate ones). Also, they asked for > > some > > > help by Flink committers / contributors to help with the reviewing of > > > incoming contributions. > > > I'm definitively willing to help out to give the community there a good > > > start. > > > > > > > > > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html > > > > > > > > > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels <m...@apache.org> > > > wrote: > > > > > >> Hi Robert, > > >> > > >> We had this discussion before when I suggested to use an external > > >> repository to manage connectors. Ever since I have come to the > > >> conclusion that the overhead of maintaining two source repositories > > >> along with maintaining code and integration, documentation, and CI, is > > >> not worth the effort. Thanks for bringing up the discussion again. > > >> > > >> I think the most important issues with moving connectors out of the > > >> core repository are > > >> > > >> 1) Maintenance > > >> Connectors need to be maintained. Otherwise, they are worthless. > > >> > > >> 2) Plugability > > >> Connectors need to be easy to use for the user. Otherwise, nobody will > > >> use them. > > >> > > >> What is the difference between #1, #3 and #4? I don't see a difference > > >> except that #3 and #4 are central repositories. Still, maintenance and > > >> plugability are very unclear. > > >> > > >> Only #2 is left :) Apache Bahir looks like a promising project. Let's > > >> see how the community reacts. > > >> > > >> Cheers, > > >> Max > > >> > > >> > > >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetz...@apache.org > > > > >> wrote: > > >> > Thank you for your responses. I will get in touch with the Bahir > > >> community > > >> > to see what they are thinking about this. > > >> > Once we know a bit more about the details of such a collaboration, > we > > >> can > > >> > make a final decision here. > > >> > > > >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrm...@apache.org > > > > >> wrote: > > >> > > > >> >> I agree with Stephan that the main problem is maintenance overhead > > for > > >> the > > >> >> Flink community. If we could maintain all connectors ourselves then > > >> there > > >> >> would not be an immediate need to out source the connectors. Thus, > > the > > >> >> solution should reduce the workload for the core project. > > >> >> > > >> >> Personally, I would favour option #2 if Apache Bahir is willing to > > add > > >> >> Flink as another supported streaming platform. This would give the > > >> >> streaming connectors a more prominent home than the personal > > >> repository of > > >> >> a contributor. > > >> >> > > >> >> Furthermore, contributing the connectors to Apache Bahir entails > that > > >> the > > >> >> connector artifacts are deployed to a central repository (I > assume). > > >> That > > >> >> way one can more easily add them to one's project instead of having > > to > > >> >> build and install the code yourself. > > >> >> > > >> >> I'm also not sure what happens to a public github repository which > > has > > >> not > > >> >> been forked and is then deleted. I would assume that the content is > > >> then > > >> >> lost. > > >> >> > > >> >> To create a "flink-connectors" repository independent of Flink > sounds > > >> quite > > >> >> similar to creating another Apache Bahir. I think it would be > better > > to > > >> >> join forces with the existing Bahir community if possible. > > >> >> > > >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <se...@apache.org> > > wrote: > > >> >> > > >> >> > Thanks Robert for bringing this up. > > >> >> > > > >> >> > I think that "flink-contrib" will not really solve the problem, > > >> because > > >> >> the > > >> >> > connectors would still have to be maintained by the core > community, > > >> we > > >> >> > would need to guarantee test stability. It will be to a large > > extend > > >> >> > similar to adding them to "flink-streaming-connectors", except > that > > >> we > > >> >> may > > >> >> > declare via the artifact name that the quality is not guaranteed > to > > >> be > > >> >> too > > >> >> > high. > > >> >> > > > >> >> > Regarding moving the code to a separate repository: I think that > > also > > >> >> only > > >> >> > solves the problem, if that repository is organizationally > > different > > >> from > > >> >> > the core clink repository. Different release cycles, different > > >> >> maintainers. > > >> >> > > > >> >> > > > >> >> > The quintessence is for me that the connectors need to be handled > > by > > >> a > > >> >> > different organization than the core flink community. > > >> >> > That would leave us with > > >> >> > - The contributors' own repository > > >> >> > - Apache Bahir > > >> >> > - An organizationally different "flink-connectors" repository, > > >> possibly > > >> >> > with different committers / PMC > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai < > > >> tzuli...@gmail.com> > > >> >> > wrote: > > >> >> > > > >> >> > > Good point. Discussion for each new connector is also an > overhead > > >> we > > >> >> > should > > >> >> > > avoid. > > >> >> > > > > >> >> > > IMHO, option #2 doesn’t seem like a reasonable staging area. > Say > > we > > >> >> > decide > > >> >> > > we’d like to move a connector from Bahir to Flink in the > future, > > >> >> there’ll > > >> >> > > be two of the connector in separate Apache projects. > Personally I > > >> think > > >> >> > > option #2 is more like a way to go if some day we’d like to > > migrate > > >> >> some > > >> >> > of > > >> >> > > our currently supported connectors in Flink to Bahir (perhaps > an > > >> >> > > alternative to the discussion in "Externalizing the Flink > > >> connectors”). > > >> >> > > > > >> >> > > Defining option #1 as a staging area seems nice; the > contributor > > >> >> notifies > > >> >> > > the Flink community of the work by adding it to the 3rd party > > >> packages > > >> >> > > page, and in the future we can contact the contributor to ask > > >> whether > > >> >> > > they’d like to migrate the connector to Flink when we see more > > user > > >> >> > demand > > >> >> > > and committer support. > > >> >> > > > > >> >> > > With this approach, do you think we should assume that future > new > > >> >> > > connectors always go to option #1 first? If not, I would > assume > > >> that > > >> >> in > > >> >> > > the future, Flink JIRAs for new connectors are only opened if > > we’d > > >> like > > >> >> > to > > >> >> > > add them directly to Flink (perhaps the contribution guideline > > can > > >> link > > >> >> > to > > >> >> > > Flink development Roadmaps like [1] as a reference to > connectors > > >> that > > >> >> the > > >> >> > > community has already considered to support?). > > >> >> > > > > >> >> > > [1] > > >> >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JK > Xm- > > >> >> > > 778DAD7eqw5GANwE/edit# > > >> >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5J > KXm- > > >> >> > > 778DAD7eqw5GANwE/edit> > > >> >> > > > > >> >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger ( > > >> rmetz...@apache.org) > > >> >> > > wrote: > > >> >> > > > > >> >> > > Hi Gordon, > > >> >> > > thank you for your response. > > >> >> > > > > >> >> > > I agree with your observation that some "staging" area is > helpful > > >> to > > >> >> test > > >> >> > > how many contributors / users are interested in a connector. > But > > I > > >> >> wonder > > >> >> > > if #1 or #2 can also serve as a staging area: As soon as we see > > >> that > > >> >> > there > > >> >> > > is a lot of interest in a connector, we add it to the main > > >> >> > > "flink-*-connectors" directory. > > >> >> > > Having too many ways to contribute connectors might make the > > >> >> discussions > > >> >> > > for each new connector quite complicated. > > >> >> > > > > >> >> > > You are right, the intention of the "Externalizing the Flink > > >> >> connectors” > > >> >> > > discussion was more about the release intervals, test time etc. > > >> >> > > > > >> >> > > Regards, > > >> >> > > Robert > > >> >> > > > > >> >> > > > >> >> > > >> > > > > > > > > >
