+1 to the goal, and replying inline on details. On Mon, Oct 9, 2017 at 8:06 AM, Aljoscha Krettek <aljos...@apache.org> wrote:
> > - We want to update the Flink dependencies to _2.11 dependencies because > 2.10 is quite outdated - This doesn't work well because some modules (examples, for example) > depend on all Runners and at least the Spark Runner has _2.10 dependencies > Let's expedite making this possible, and circle back to getting the build to an ideal state after unblocking this very reasonable change. It is not reasonable for any runner dependencies in Beam to be coupled, certainly not Scala version. We've been monolithic so far because it is easier to manage, but it was never a long term solution. It will mean that the examples cannot have -P flink-runner and -P spark-runner at the same time. But my position is that we should never expect to be able to use two such profiles at the same time. Of course, if it is technically feasible to transition both runners (I don't really know about Spark's coupling with Scala versions) that is event easier and defers the larger issue for a bit. I see two solutions for this: > - Introducing a project-wide Scala version property > - Allowing differing Scala versions for different runners, ensure that we > never have a situation where we have several Runners as dependency. For the > the "Maven Install" pre-commit hook this could mean splitting it up per > runner. > I support the latter regardless. I want separate configurations for separate runners, fully embracing the fact that they can diverge. We already intend to split up the build to be much more fine grained. The only reason we have a monolithic precommit is Maven's extraordinary lack of support for any other way of building. We have essentially started to build something akin to a particular invocation of "make" in the form of interrelated Jenkins jobs to work around Maven's limitations [1]. Until we can get that approach working at 100% I am fine with splitting builds aggressively. You will find that it is quite hard not to duplicate a lot of work when splitting them, unless we abandon the Jenkins plugin and use a sequence of carefully crafted shell commands. Kenn [1] https://github.com/apache/beam/blob/master/.test-infra/jenkins/PreCommit_Pipeline.groovy
