I think it should not be at least the flink-dist but exactly the remaining flink-dist module. Otherwise we do redundant work.
On Wed, Mar 15, 2017 at 5:03 PM, Robert Metzger <rmetz...@apache.org> wrote: > "flink-core" means the main repository, not the "flink-core" module. > > When doing a release, we need to build the flink main code first, because > the flink-libraries depend on that. > Once the "flink-libraries" are build, we need to run the main build again > (at least the flink-dist module), so that it is pulling the artifacts from > the flink-libraries to put them into the opt/ folder of the final artifact. > > > > > On Wed, Mar 15, 2017 at 4:44 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > > > I'm ok with point 3. > > > > Concerning point 8: Why do we have to build flink-core twice after having > > it built as a dependency for flink-libraries? This seems wrong to me. > > > > Cheers, > > Till > > > > On Wed, Mar 15, 2017 at 4:23 PM, Robert Metzger <rmetz...@apache.org> > > wrote: > > > > > Thank you. Running on AWS is a good idea! > > > Let me know if you (or anybody else) wants to help me with the > > > infrastructure work! Any help is much appreciated (as I've said > before, I > > > don't really have time for doing this, but it has to be done :) ) > > > > > > I'm against creating two new repositories. I fear that this introduces > > too > > > much complexity and too many repositories. > > > "flink" and "flink-libraries" are hopefully enough to get the build > time > > > significantly down. > > > We can also consider putting the connectors into the "flink-libraries" > > repo > > > if we need to further reduce the build time. > > > > > > We should probably move "flink-table" of out "flink-libraries" if we > want > > > to keep "flink-table" in the main repo. (This would eliminate the > > > "flink-libraries" module from main. > > > > > > Also, I agree that "flink-statebackend-rocksdb" is not correctly placed > > in > > > contrib anymore. > > > > > > > > > On Wed, Mar 15, 2017 at 4:07 PM, Greg Hogan <c...@greghogan.com> > wrote: > > > > > > > Robert, appreciate your kickstarting this task. > > > > > > > > We should compare the verification time with and without the listed > > > > modules. I’ll try to run this by tomorrow on AWS and on Travis. > > > > > > > > Should we maintain separate repos for flink-contrib and > > flink-libraries? > > > > Are you intending that we move flink-table out of flink-libraries > (and > > > > perhaps flink-statebackend-rocksdb out of flink-contrib)? > > > > > > > > Greg > > > > > > > > > > > > > On Mar 15, 2017, at 9:55 AM, Robert Metzger <rmetz...@apache.org> > > > wrote: > > > > > > > > > > Thank you for looking into this Till. > > > > > > > > > > I think we then have to split the repositories. > > > > > My main motivation for doing this is that it seems to be the only > > > > feasible > > > > > way of scaling the community to allow more committers working on > the > > > > > libraries. > > > > > > > > > > I'll take care of getting things started. > > > > > > > > > > As the next steps I propose to: > > > > > 1. Ask INFRA to rename https://git-wip-us.apache.org/ > > > repos/asf?p=flink- > > > > > connectors.git;a=summary to "flink-libraries" > > > > > 2. Ask INFRA to set up GitHub and travis integration for > > > > "flink-libraries" > > > > > 3. Put the code of "flink-ml", "flink-gelly", "flink-python", > > > > "flink-cep", > > > > > "flink-scala-shell", "flink-storm" into the new repository. (I > > decided > > > > > against moving flink-contrib there, because rocksdb is in the > contrib > > > > > module, for flink-table, I'm undecided, but I kept it in the main > > repo > > > > > because its probably going to interact more with the core code in > the > > > > > future) > > > > > I try to preserve the history of those modules when splitting them > > into > > > > the > > > > > new repo > > > > > 4. I'll close all pull requests against those modules in the main > > repo. > > > > > 5. I'll set up a minimal documentation page for the library > > repository, > > > > > similar to the main documentation. > > > > > 6. I'll update the documentation build process to build both > > > > documentations > > > > > & link them to each other > > > > > 7. I'll update the nightly deployment process to include both > > > > repositories > > > > > 8. I'll update the release script to create the Flink release out > of > > > both > > > > > repositories. In order to put the libraries into the opt/ dir of > the > > > > > release, I'll need to change the build of "flink-dist" so that it > > first > > > > > builds flink core, then the libraries and then the core again with > > the > > > > > libraries as an additional dependency. > > > > > > > > > > The main question for the community is: do you agree with point 3 ? > > > Would > > > > > you like to include more or less? > > > > > > > > > > I'll start with 1. and 2. tomorrow morning. > > > > > > > > > > > > > > > > > > > > On Wed, Mar 15, 2017 at 1:48 PM, Till Rohrmann < > trohrm...@apache.org > > > > > > > wrote: > > > > > > > > > >> In theory we could have a merging bot which solves the problem of > > the > > > > >> "commit window". Once the PR passes all tests and has enough +1s, > > the > > > > bot > > > > >> could do the merging and, thus, it effectively linearizes the > merge > > > > >> process. > > > > >> > > > > >> I think the second point is actually a disadvantage because there > is > > > not > > > > >> such an immediate incentive/pressure to fix the broken module if > it > > > > lives > > > > >> in a separate repository. Furthermore, breaking API changes in the > > > core > > > > >> will most likely go unnoticed for some time in other modules which > > are > > > > not > > > > >> developed so actively. In the worst case these things will only be > > > > noticed > > > > >> when we try to make a release. > > > > >> > > > > >> But I also agree that we are not Google and we don't have the > > > > capacities to > > > > >> maintain such a smooth a build process that we can keep all the > code > > > in > > > > a > > > > >> single repository. > > > > >> > > > > >> I looked a bit into Gradle and as far as I can tell it offers some > > > nice > > > > >> features wrt incrementally building projects. This would be > > beneficial > > > > for > > > > >> local development but it would not solve our build time problems > on > > > > Travis. > > > > >> Gradle intends to introduce a task result cache which allows to > > reuse > > > > >> results across builds. This could help when building on Travis, > > > > however, it > > > > >> is not yet fully implemented. Moreover, migrating from Maven to > > Gradle > > > > >> won't come for free (there's simply no free lunch out there) and > we > > > > might > > > > >> risk to introduce new bugs. Therefore, I would vote to split the > > > > repository > > > > >> in order to mitigate our current problems with Travis and the > build > > > > time in > > > > >> general. Whether to use a different build system or not can then > be > > > > >> discussed as an orthogonal question. > > > > >> > > > > >> Cheers, > > > > >> Till > > > > >> > > > > >> On Tue, Mar 14, 2017 at 8:05 PM, Stephan Ewen <se...@apache.org> > > > wrote: > > > > >> > > > > >>> Some other thoughts on how repository split would help. I am not > > sure > > > > for > > > > >>> all of them, so please comment: > > > > >>> > > > > >>> - There is less competition for a "commit window". It happens a > > lot > > > > >>> already that you run all tests and want to commit, but there was > a > > > > commit > > > > >>> in the meantime. You rebase, need to re-test, again commit in the > > > > >> meantime. > > > > >>> For a "linear" commit history, this may become a bottleneck > > > > >> eventually > > > > >>> as well. > > > > >>> > > > > >>> - There is less risk of broken master. If one repository/modules > > > > breaks > > > > >>> its master, the others can still continue. > > > > >>> > > > > >>> Stephan > > > > >>> > > > > >>> > > > > >>> On Fri, Mar 10, 2017 at 12:20 PM, Till Rohrmann < > > > trohrm...@apache.org> > > > > >>> wrote: > > > > >>> > > > > >>>> Thanks for all your input. In order to wrap the discussion up > I'd > > > like > > > > >> to > > > > >>>> summarize the mentioned points: > > > > >>>> > > > > >>>> The problem of increasing build times and complexity of the > > project > > > > has > > > > >>>> been acknowledged. Ideally we would have everything in one > > > repository > > > > >>> using > > > > >>>> an incremental build tool. Since Maven does not properly support > > > this > > > > >> we > > > > >>>> would have to switch our build tool to something like Gradle, > for > > > > >>> example. > > > > >>>> > > > > >>>> Another option is introducing build profiles for different sets > of > > > > >>> modules > > > > >>>> as well as separating integration and unit tests. The third > > > > alternative > > > > >>>> would be creating sub-projects with their own repositories. I > > > actually > > > > >>>> think that these two proposal are not necessarily exclusive and > it > > > > >> would > > > > >>>> also make sense to have a separation between unit and > integration > > > > tests > > > > >>> if > > > > >>>> we split the respository. > > > > >>>> > > > > >>>> The overall consensus seems to be that we don't want to split > the > > > > >>> community > > > > >>>> and want to keep everything under the same umbrella. I think > this > > is > > > > >> the > > > > >>>> right way to go, because otherwise some parts of the project > could > > > > >> become > > > > >>>> second class citizens. Given that and that we continue using > > Maven, > > > I > > > > >>> still > > > > >>>> think that creating sub-projects for the libraries, for example, > > > could > > > > >> be > > > > >>>> beneficial. A split could reduce the project's complexity and > make > > > it > > > > >>>> potentially easier for libraries to get actively developed. The > > main > > > > >>>> concern is setting up the build infrastructure to aggregate docs > > > from > > > > >>>> multiple repositories and making them publicly available. > > > > >>>> > > > > >>>> Since I started this thread and I would really like to see > Flink's > > > ML > > > > >>>> library being revived again, I'd volunteer investigating first > > > whether > > > > >> it > > > > >>>> is doable establishing a proper incremental build for Flink. If > > that > > > > >>> should > > > > >>>> not be possible, I will look into splitting the repository, > first > > > only > > > > >>> for > > > > >>>> the libraries. I'll share my results with the community once I'm > > > done > > > > >>> with > > > > >>>> the investigation. > > > > >>>> > > > > >>>> Cheers, > > > > >>>> Till > > > > >>>> > > > > >>>> On Fri, Feb 24, 2017 at 3:50 PM, Robert Metzger < > > > rmetz...@apache.org> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> @Jin Mingjian: You can not use the paid travis version for open > > > > >> source > > > > >>>>> projects. It only works for private repositories (at least back > > > then > > > > >>> when > > > > >>>>> we've asked them about that). > > > > >>>>> > > > > >>>>> @Stephan: I don't think that incremental builds will be > available > > > > >> with > > > > >>>>> Maven anytime soon. > > > > >>>>> > > > > >>>>> I agree that we need to fix the build time issue on Travis. > I've > > > > >>> recently > > > > >>>>> pushed a commit to use now three instead of two test groups. > > > > >>>>> But I don't think that this is feasible long-term solution. > > > > >>>>> > > > > >>>>> If this discussion is only about reducing the build and test > > time, > > > > >>>>> introducing build profiles for different components as Aljoscha > > > > >>> suggested > > > > >>>>> would solve the problem Till mentioned. > > > > >>>>> Also, if we decide that travis is not a good tool anymore for > the > > > > >>>> testing, > > > > >>>>> I guess we can find a different solution. There are now > > competitors > > > > >> to > > > > >>>>> Travis that might be willing to offer a paid plan for an open > > > source > > > > >>>>> project, or we set up our own infra on a server sponsored by > one > > of > > > > >> the > > > > >>>>> contributing companies. > > > > >>>>> If we want to solve "community issues" with the change as well, > > > then > > > > >> I > > > > >>>>> think its work the effort of splitting up Flink into different > > > > >>>>> repositories. > > > > >>>>> > > > > >>>>> Splitting up repositories is not a trivial task in my opinion. > As > > > > >>> others > > > > >>>>> have mentioned before, we need to consider the following > things: > > > > >>>>> - How are we doing to build the documentation? Ideally every > repo > > > > >>> should > > > > >>>>> contain its docs, so we would need to pull them together when > > > > >> building > > > > >>>> the > > > > >>>>> main docs. > > > > >>>>> - How do organize the dependencies? If we have library > repository > > > > >>> depend > > > > >>>> on > > > > >>>>> snapshot Flink versions, we need to make sure that the snapshot > > > > >>>> deployment > > > > >>>>> always works. This also means that people working on a library > > > > >>> repository > > > > >>>>> will pull from snapshot OR need to build first locally. > > > > >>>>> - We need to update the release scripts > > > > >>>>> > > > > >>>>> If we commit to do these changes, we need to assign at least > one > > > > >>>> committer > > > > >>>>> (yes, in this case we need somebody who can commit, for example > > for > > > > >>>>> updating the buildbot stuff) who volunteers to do the change. > > > > >>>>> I've done a lot of infrastructure work in the past, but I'm > > > currently > > > > >>>>> pretty booked with many other things, so I don't realistically > > see > > > > >>> myself > > > > >>>>> doing that. Max who used to work on these things is taking some > > > time > > > > >>> off. > > > > >>>>> I think we need, best case 3 days for the change, worst case 5 > > > days. > > > > >>> The > > > > >>>>> problem is that there are no "unit tests" for the infra stuff, > so > > > > >> many > > > > >>>>> things are "trial and error" (like Apache's buildbot, our > release > > > > >>>> scripts, > > > > >>>>> the doc scripts, maven stuff, nightly builds). > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> On Thu, Feb 23, 2017 at 1:33 PM, Stephan Ewen < > se...@apache.org> > > > > >>> wrote: > > > > >>>>> > > > > >>>>>> If we can get a incremental builds to work, that would > actually > > be > > > > >>> the > > > > >>>>>> preferred solution in my opinion. > > > > >>>>>> > > > > >>>>>> Many companies have invested heavily in making a "single > > > > >> repository" > > > > >>>> code > > > > >>>>>> base work, because it has the advantage of not having to > > > > >>> update/publish > > > > >>>>>> several repositories first. > > > > >>>>>> However, the strong prerequisite for that is an incremental > > build > > > > >>>> system > > > > >>>>>> that builds only (fine grained) what it has to build. I am not > > > sure > > > > >>> how > > > > >>>>> we > > > > >>>>>> could make that work > > > > >>>>>> with Maven and Travis... > > > > >>>>>> > > > > >>>>>> On Wed, Feb 22, 2017 at 10:42 PM, Greg Hogan < > > c...@greghogan.com> > > > > >>>> wrote: > > > > >>>>>> > > > > >>>>>>> An additional option for reducing time to build and test is > > > > >>> parallel > > > > >>>>>>> execution. This would help users more than on TravisCI since > > > > >> we're > > > > >>>>>>> generally running on multi-core machines rather than VM > slices. > > > > >>>>>>> > > > > >>>>>>> Is the idea that each user would only check out the modules > > that > > > > >> he > > > > >>>> or > > > > >>>>>> she > > > > >>>>>>> is developing with? For example, if a developer is not > working > > on > > > > >>>>>>> flink-mesos or flink-yarn then the "flink-deploy" module > would > > > > >> not > > > > >>> be > > > > >>>>>> clone > > > > >>>>>>> to their filesystem? > > > > >>>>>>> > > > > >>>>>>> We can run a TravisCI nightly build on each repo to validate > > > > >>> against > > > > >>>>> API > > > > >>>>>>> changes. > > > > >>>>>>> > > > > >>>>>>> Greg > > > > >>>>>>> > > > > >>>>>>> On Wed, Feb 22, 2017 at 12:24 PM, Fabian Hueske < > > > > >> fhue...@gmail.com > > > > >>>> > > > > >>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Hi everybody, > > > > >>>>>>>> > > > > >>>>>>>> I think this should be a discussion about the benefits and > > > > >>>> drawbacks > > > > >>>>> of > > > > >>>>>>>> separating the code into distinct repositories from a > > > > >> development > > > > >>>>> point > > > > >>>>>>> of > > > > >>>>>>>> view. > > > > >>>>>>>> So I agree with Stephan that we should not divide the > > community > > > > >>> by > > > > >>>>>>> creating > > > > >>>>>>>> separate groups of committers. > > > > >>>>>>>> Also the discussion about independent releases is not be > > > > >> strictly > > > > >>>>>> related > > > > >>>>>>>> to the decision, IMO. > > > > >>>>>>>> > > > > >>>>>>>> I see a few pros and cons for splitting the code base into > > > > >>> separate > > > > >>>>>>>> repositories which (I think) haven't been mentioned before: > > > > >>>>>>>> pros: > > > > >>>>>>>> - IDE setup will be leaner. It is not necessary to compile > the > > > > >>>> whole > > > > >>>>>> code > > > > >>>>>>>> base to run a test after switching a branch. > > > > >>>>>>>> cons: > > > > >>>>>>>> - developing libraries features that require changes in the > > > > >> core > > > > >>> / > > > > >>>>> APIs > > > > >>>>>>>> become more time consuming due to back-and-forth between > code > > > > >>>> bases. > > > > >>>>>>>> However, I think this is not very often the case. > > > > >>>>>>>> > > > > >>>>>>>> Aljoscha has good points as well. Many of the build issues > > > > >> could > > > > >>> be > > > > >>>>>>> solved > > > > >>>>>>>> by different build profiles and configurations. > > > > >>>>>>>> > > > > >>>>>>>> Best, Fabian > > > > >>>>>>>> > > > > >>>>>>>> 2017-02-22 14:59 GMT+01:00 Gábor Hermann < > > > > >> m...@gaborhermann.com > > > > >>>> : > > > > >>>>>>>> > > > > >>>>>>>>> @Stephan: > > > > >>>>>>>>> > > > > >>>>>>>>> Although I tried to raise some issues about splitting > > > > >>> committers, > > > > >>>>> I'm > > > > >>>>>>>>> still strongly in favor of some kind of restructuring. We > > > > >> just > > > > >>>> have > > > > >>>>>> to > > > > >>>>>>> be > > > > >>>>>>>>> conscious about the disadvantages. > > > > >>>>>>>>> > > > > >>>>>>>>> Not splitting the committers could leave the libraries in > the > > > > >>>> same > > > > >>>>>>>>> stalling status, described by Till. Of course, dedicating > > > > >>> current > > > > >>>>>>>>> committers as shepherds of the libraries could easily > resolve > > > > >>> the > > > > >>>>>>> issue. > > > > >>>>>>>>> But that requires time from current committers. It seems > like > > > > >>>>>>> trade-offs > > > > >>>>>>>>> between code quality, speed of development, and committer > > > > >>>> efforts. > > > > >>>>>>>>> > > > > >>>>>>>>> From what I see in the discussion about ML, there are many > > > > >>> people > > > > >>>>>>> willing > > > > >>>>>>>>> to contribute as well as production use-cases. This means > we > > > > >>>> could > > > > >>>>>> and > > > > >>>>>>>>> should move forward. However, the development speed is > > > > >>>>> significantly > > > > >>>>>>>> slowed > > > > >>>>>>>>> down by stalling PRs. The proposal for contributors helping > > > > >> the > > > > >>>>>> review > > > > >>>>>>>>> process did not really work out so far. In my opinion, > either > > > > >>>> code > > > > >>>>>>>> quality > > > > >>>>>>>>> (by more easily accepting new committers) or some committer > > > > >>> time > > > > >>>>>>>>> (reviewing/merging) should be sacrificed to move forward. > As > > > > >>> Till > > > > >>>>> has > > > > >>>>>>>>> indicated, it would be shameful if we let this contribution > > > > >>>> effort > > > > >>>>>> die. > > > > >>>>>>>>> > > > > >>>>>>>>> Cheers, > > > > >>>>>>>>> Gabor > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
