Thanks a lot for your summary Chesnay. I agree with you that we have no consensus in the community for splitting up the repository immediately, and I agree with you that we should have a separate discussion about reducing the build time (which is already making good progress).
Also, I will keep the thoughts about decentralising the Flink development in the back of my head and bring it up again whenever I feel it's the right time. On Wed, Aug 14, 2019 at 2:26 PM Chesnay Schepler <ches...@apache.org> wrote: > Let's recap a bit: > > Several people have raised the argument that build times can be kept in > check via other means (mostly differential builds via some means, be it > custom scripts or switching to gradle). I will start a separate > discussion thread on this topic, since it is a useful discussion in any > case. > I agree with this, and believe it is feasible to update the CI process > to behave as if the repository was split. > > > The suggestion of a "project split" within a single repository was > brought up. > This approach is a mixed bag; it avoids the downsides to the development > process that multiple repositories would incur, but also only has few > upsides. It seems primarily relevant for local development, where one > might want to skip certain modules when running tests. > > There's no benefit from the CI side: since we're still limited to 1 > .travis.yml, whatever rules we want to set up (e.g., "do not test core > if only connectors are modified") have to be handled by the CI scripts > regardless of whether the project is split or not. > > Overall, I'd like to put this item on ice for the time being; the > subsequent item is related, vastly more impactful and may also render > this item obsolete. > > > A major topic of discussion is that of the development process. It was > pointed how that having a split repository makes the dev process more > complicated, since certain changes turn into a 2 step process (merge to > core, then merge to connectors). Others have pointed out that this may > actually be an advantage, as it (to some extent) enforces that changes > to core are also tested in core. > > I find myself more in the latter camp; it is all to easy for people to > make a change to the core while making whatever adjustments to > connectors to make things fit. A recent change to the ClosureCleaner in > 1.8.0 <https://issues.apache.org/jira/browse/FLINK-13586> comes to mind, > which, with a split repo, may have resulted in build failures in the > connectors project. (provided that the time-frame between the 2 merges > is sufficiently large...) As Arvid pointed out, having to feel the pain > that users have to go through may not be such a bad thing. > > This is a fundamental discussion as to whether we want to continue with > a centralized development of all components. > > Robert also pointed out that such a split could result in us > establishing entirely separate projects. We've had times in the past > (like the first flink-ml library) where such a setup may have simplified > things (back then we had lot's of contributors but no committer to > shepherd the effort; a separate project could be more lenient when it > comes to appointing new committers). > > > @Robert We should have a SNAPSHOT dependency /somewhere/ in the > connector repo, to detect issues (like the ClosureCleaner one) in a > timely manner and to prepare for new features so that we can have a > timely release after core, but not necessarily on the master branch. > > @Bowen I have implemented and deployed your suggestion to cancel Travis > builds if the associated PR has been closed. > > > On 07/08/2019 13:14, Chesnay Schepler wrote: > > Hello everyone, > > > > The Flink project sees an ever-increasing amount of dev activity, both > > in terms of reworked and new features. > > > > This is of course an excellent situation to be in, but we are getting > > to a point where the associate downsides are becoming increasingly > > troublesome. > > > > The ever increasing build times, in addition to unstable tests, > > significantly slow down the develoment process. > > Additionally, pull requests for smaller features frequently slip > > through the crasks as they are being buried under a mountain of other > > pull requests. > > > > As a result I'd like to start a discussion on splitting the Flink > > repository. > > > > In this mail I will outline the core idea, and what problems I > > currently envision. > > > > I'd specifically like to encourage those who were part of similar > > initiatives in other projects to share the experiences and ideas. > > > > > > General Idea > > > > For starters, the idea is to create a new repository for > > "flink-connectors". > > For the remainder of this mail, the current Flink repository is > > referred to as "flink-main". > > > > There are also other candidates that we could discuss in the future, > > like flink-libraries (the next top-priority repo to ease flink-ml > > development), metric reporters, filesystems and flink-formats. > > > > Moving out flink-connectors provides the most benefits, as we straight > > away save at-least an hour of testing time, and not being included in > > the binary distribution simplifies a few things. > > > > > > Problems to solve > > > > To make this a reality there's a number of questions we have to > > discuss; some in the short-term, others in the long-term. > > > > 1) Git history > > > > We have to decide whether we want to rewrite the history of sub > > repositories to only contain diffs/commits related to this part of > > Flink, or whether we just fork from some commit in flink-main and > > add a commit to the connector repo that "transforms" it from > > flink-main to flink-connectors (i.e., remove everything unrelated to > > connectors + update module structure etc.). > > > > The latter option would have the advantage that our commit book > > keeping in JIRA would still be correct, but it would create a > > significant divide between the current and past state of the > > repository. > > > > 2) Maven > > > > We should look into whether there's a way to share dependency/plugin > > configurations and similar, so we don't have to keep them in sync > > manually across multiple repositories. > > > > A new parent Flink pom that all repositories define as their parent > > could work; this would imply splicing out part of the current room > > pom.xml. > > > > 3) Documentation > > > > Splitting the repository realistically also implies splitting the > > documentation source files (At the beginning we can get by with > > having it still in flink-main). > > We could just move the relevant files to the respective repository > > (while maintaining the directory structure), and merge them when > > building the docs. > > > > We also have to look at how we can handle java-/scaladocs; e.g. > > whether it is possible to aggregate them across projects. > > > > 4) CI (end-to-end tests) > > > > The very basic question we have to answer is whether we want E2E > > tests in the sub repositories. If so, we need to find a way to share > > e2e-tooling. > > > > 5) Releases > > > > We have to discuss how our release process will look like. This may > > also have repercussions on how repositories may depend on each other > > (SNAPSHOT vs LATEST). Note that this should be discussed for each > > repo separately. > > > > The current options I see are the following: > > > > a) Single release > > > > Release all repositories at once as a single product. > > > > The source release would be a collection of repositories, like > > flink/ > > |--flink-main/ > > |--flink-core/ > > |--flink-runtime/ > > ... > > |--flink-connectors/ > > ... > > |--flink-.../ > > ... > > > > This option requires a SNAPSHOT dependency between Flink > > repositories, but it is pretty much how things work at the moment. > > > > b) Synced releases > > > > Similar to a), except that each repository gets their own source > > release that they may released independent of other repositories. > > For a given release cycle each repo would produce exactly one > > release. > > > > This option requires a SNAPSHOT dependency between Flink > > repositories. Once any repositories has created an RC or > > finished it's release, release-branches in other repos can > > switch to that version. > > > > This approach is a tad more flexible than a), but requires more > > coordination between the repos. > > > > c) Separate releases > > > > Just like we handle flink-shaded; entirely separate release > > cycles; some repositories may have more releases in a given time > > period than others. > > > > This option implies a LATEST dependency between Flink > > repositories. > > > > Note that hybrid approaches would also make sense, like doing b) for > > major versions and c) for bugfix releases. > > > > For something like flink-libraries this question may also have > > repercussions on how/whether they are bundled in the distribution; > > options a)/b) would maintain the status-quo, c) and hybrid > > approaches will likely necessitate the exclusion from the > > distribution. > > > > > >
