@Jin Mingjian: You can not use the paid travis version for open source projects. It only works for private repositories (at least back then when we've asked them about that).
@Stephan: I don't think that incremental builds will be available with Maven anytime soon. I agree that we need to fix the build time issue on Travis. I've recently pushed a commit to use now three instead of two test groups. But I don't think that this is feasible long-term solution. If this discussion is only about reducing the build and test time, introducing build profiles for different components as Aljoscha suggested would solve the problem Till mentioned. Also, if we decide that travis is not a good tool anymore for the testing, I guess we can find a different solution. There are now competitors to Travis that might be willing to offer a paid plan for an open source project, or we set up our own infra on a server sponsored by one of the contributing companies. If we want to solve "community issues" with the change as well, then I think its work the effort of splitting up Flink into different repositories. Splitting up repositories is not a trivial task in my opinion. As others have mentioned before, we need to consider the following things: - How are we doing to build the documentation? Ideally every repo should contain its docs, so we would need to pull them together when building the main docs. - How do organize the dependencies? If we have library repository depend on snapshot Flink versions, we need to make sure that the snapshot deployment always works. This also means that people working on a library repository will pull from snapshot OR need to build first locally. - We need to update the release scripts If we commit to do these changes, we need to assign at least one committer (yes, in this case we need somebody who can commit, for example for updating the buildbot stuff) who volunteers to do the change. I've done a lot of infrastructure work in the past, but I'm currently pretty booked with many other things, so I don't realistically see myself doing that. Max who used to work on these things is taking some time off. I think we need, best case 3 days for the change, worst case 5 days. The problem is that there are no "unit tests" for the infra stuff, so many things are "trial and error" (like Apache's buildbot, our release scripts, the doc scripts, maven stuff, nightly builds). On Thu, Feb 23, 2017 at 1:33 PM, Stephan Ewen <se...@apache.org> wrote: > If we can get a incremental builds to work, that would actually be the > preferred solution in my opinion. > > Many companies have invested heavily in making a "single repository" code > base work, because it has the advantage of not having to update/publish > several repositories first. > However, the strong prerequisite for that is an incremental build system > that builds only (fine grained) what it has to build. I am not sure how we > could make that work > with Maven and Travis... > > On Wed, Feb 22, 2017 at 10:42 PM, Greg Hogan <c...@greghogan.com> wrote: > > > An additional option for reducing time to build and test is parallel > > execution. This would help users more than on TravisCI since we're > > generally running on multi-core machines rather than VM slices. > > > > Is the idea that each user would only check out the modules that he or > she > > is developing with? For example, if a developer is not working on > > flink-mesos or flink-yarn then the "flink-deploy" module would not be > clone > > to their filesystem? > > > > We can run a TravisCI nightly build on each repo to validate against API > > changes. > > > > Greg > > > > On Wed, Feb 22, 2017 at 12:24 PM, Fabian Hueske <fhue...@gmail.com> > wrote: > > > > > Hi everybody, > > > > > > I think this should be a discussion about the benefits and drawbacks of > > > separating the code into distinct repositories from a development point > > of > > > view. > > > So I agree with Stephan that we should not divide the community by > > creating > > > separate groups of committers. > > > Also the discussion about independent releases is not be strictly > related > > > to the decision, IMO. > > > > > > I see a few pros and cons for splitting the code base into separate > > > repositories which (I think) haven't been mentioned before: > > > pros: > > > - IDE setup will be leaner. It is not necessary to compile the whole > code > > > base to run a test after switching a branch. > > > cons: > > > - developing libraries features that require changes in the core / APIs > > > become more time consuming due to back-and-forth between code bases. > > > However, I think this is not very often the case. > > > > > > Aljoscha has good points as well. Many of the build issues could be > > solved > > > by different build profiles and configurations. > > > > > > Best, Fabian > > > > > > 2017-02-22 14:59 GMT+01:00 Gábor Hermann <m...@gaborhermann.com>: > > > > > > > @Stephan: > > > > > > > > Although I tried to raise some issues about splitting committers, I'm > > > > still strongly in favor of some kind of restructuring. We just have > to > > be > > > > conscious about the disadvantages. > > > > > > > > Not splitting the committers could leave the libraries in the same > > > > stalling status, described by Till. Of course, dedicating current > > > > committers as shepherds of the libraries could easily resolve the > > issue. > > > > But that requires time from current committers. It seems like > > trade-offs > > > > between code quality, speed of development, and committer efforts. > > > > > > > > From what I see in the discussion about ML, there are many people > > willing > > > > to contribute as well as production use-cases. This means we could > and > > > > should move forward. However, the development speed is significantly > > > slowed > > > > down by stalling PRs. The proposal for contributors helping the > review > > > > process did not really work out so far. In my opinion, either code > > > quality > > > > (by more easily accepting new committers) or some committer time > > > > (reviewing/merging) should be sacrificed to move forward. As Till has > > > > indicated, it would be shameful if we let this contribution effort > die. > > > > > > > > Cheers, > > > > Gabor > > > > > > > > > > > > > >
