Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
@Lukasz: was it tested in current setup? I know groovy does it but never checked it myself. If not the vote must be conditional to that IMHO. Le 28 nov. 2017 19:19, "Robert Bradshaw" a écrit : > I also did an apache github query > > select count(*) as apache_projects, sum(uses_maven=true) as > u

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Robert Bradshaw
I also did an apache github query select count(*) as apache_projects, sum(uses_maven=true) as uses_maven, sum(uses_gradle=true) as uses_gradle from ( select repo_name, max(path contains 'pom.xml') as uses_maven, max(path contains 'gradle') as uses_gradle from [bigquery-public-data:gith

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Lukasz Cwik
Romain, Gradle has a Nexus plugin[1] which can sign and publish artifacts. Gradle also has excellent support to run Ant tasks since Ant can perform the entire release process for an ASF project. 1: https://github.com/bmuschko/gradle-nexus-plugin On Tue, Nov 28, 2017 at 9:45 AM, Scott Wegner wrot

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Scott Wegner
To add one more data point measuring general adoption of gradle vs. maven, we can look at Stackoverflow trends comparing the two tags [1]. This shows the percentage of new SO questions in a given month by tag. 'gradle' represents ~0.25% of questions, while maven is ~0.45%. So, maven is more dominan

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
Did you try a release (you can create a temporary staging repo on ASF nexus if it helps) before starting a vote? Cause you migrate and the project is no more able to release it can be a rude blocker - which never happens when needed ;). Release has a few more plugins I didn't find in gradle (can ha

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Kenneth Knowles
Yea, I think voting is the next step. Luke - I think you are obviously the right person to set up the email of what exactly we are voting on, since you've driven this improvement. On Tue, Nov 28, 2017 at 12:08 AM, Robert Bradshaw wrote: > It's great to see all the discussion going on here. > > I

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
Guys, just realized a lot of modules have threadCount=4 or so in the surefire/failsafe config. It makes it impossible to adapt the parallelism to the machine and therefore makes the parallelism inadapted and useless. Can it be a variable at least? -T1C (or -T2C) should allow to be smoother and ali

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Romain Manni-Bucau
Just to answer a previous question: An ASF github search gives me these stats: - mvn (org:apache apache filename:pom.xml path:/): 731 - gradle (org:apache apache filename:build.gradle path:/): 31 Which is consistent with what I saw in enterprises and private repos. So way different from the whol

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-28 Thread Robert Bradshaw
It's great to see all the discussion going on here. I think it's important to point out that merging a parallel set of gradle build scripts is a separate (and much less disruptive) step than, say, switching over the default (or even recommended) build/release process to use them, let alone removin

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Jean-Baptiste Onofré
Yeah, especially, I think it would have been great to have a vote before merging on master. Not a big deal, however, I'm really community focus ;) Regards JB On 11/28/2017 07:36 AM, Reuven Lax wrote: Agreed. I thinking having a formal vote before Luke had numbers and results would have been t

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Reuven Lax
Agreed. I thinking having a formal vote before Luke had numbers and results would have been too early. However now that we have such numbers, we should think about having a vote. Also, while I disagree with Romain that Gradle is not "enterprise ready" (it's heavily used by Netflix, LinkedIn, Sieme

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Jean-Baptiste Onofré
Hi Luke, just curious (and maybe I missed it): did we do a formal vote to merge the gradle build ? Gradle is now on master, we have some Jira to update the release guide with gradle. It's fine, but I remember only a discussion, not a vote. In order to embrace the community and avoid to have s

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Le 27 nov. 2017 23:07, "Eugene Kirpichov" a écrit : On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik : > > Romain, as mentioned earlier, I identified that Maven was slower because > it > > needed to finish building the entire module before depe

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Eugene Kirpichov
On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik : > > Romain, as mentioned earlier, I identified that Maven was slower because > it > > needed to finish building the entire module before dependent modules > could > > start which included running

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Lukasz Cwik
I don't believe that such a switch could happen immediately as the scope of the change is to replace 15k lines of pom files so anything which isn't gradual is unlikely to work. Below is a more thorough list of points which compare the two build systems beyond speed. Maven Java Support: Mature Pyt

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Le 27 nov. 2017 21:34, "Lukasz Cwik" a écrit : On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik : > > Romain, as mentioned earlier, I identified that Maven was slower because > it > > needed to finish building the entire module before dependen

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Ismaël Mejía
I have been a little bit out of the discussion on maven vs gradle because I was expecting the technical proof of concepts to evaluate the best approach. I deeply appreciate all the effort that Lukasz has put into the gradle version, and I also think that during the discussion Romain and others have

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Lukasz Cwik
On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik : > > Romain, as mentioned earlier, I identified that Maven was slower because > it > > needed to finish building the entire module before dependent modules > could > > start which included runnin

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
2017-11-27 20:26 GMT+01:00 Lukasz Cwik : > Romain, as mentioned earlier, I identified that Maven was slower because it > needed to finish building the entire module before dependent modules could > start which included running tests, performing checkstyle, etc... > Gradle is able to increase the pa

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Lukasz Cwik
Romain, as mentioned earlier, I identified that Maven was slower because it needed to finish building the entire module before dependent modules could start which included running tests, performing checkstyle, etc... Gradle is able to increase the parallelism of the build process since it has task

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Kenneth Knowles
Nice work, this is pretty exciting. I agree we should switch. Gradle is a great combination of being a mainstream tool and having reasonably correct model that unlocks good performance. I'm hoping to see even more improvements as we learn to use it and enable incremental builds. Kenn On Mon, Nov

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Romain Manni-Bucau
Hi Lukasz, Did you manage to identify how maven was slower and test tesla stuff and potentially a few other fixes? Side note: figures without python can be interesting cause locally - = for me - python tends to flatten the figures whereas I get something close to your conclusions without python p

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-27 Thread Lukasz Cwik
I have collected data by running several builds against master using Gradle and Maven without using Gradle's support for incremental builds. Gradle (mins) min: 25.04 max: 160.14 median: 45.78 average: 52.19 stdev: 30.80 Maven (mins) min: 56.86 max: 216.55 (actually > 240 mins because this data do

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-19 Thread Jean-Baptiste Onofré
Thanks for the update Luke. I'm updating my local working copy to do new tests. Regards JB On 11/19/2017 08:21 PM, Lukasz Cwik wrote: The gradle build rules have been merged, I'm adding a precommit[1] to start collecting data about the build times. It currently only mirrors the Java mvn instal

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-19 Thread Lukasz Cwik
The gradle build rules have been merged, I'm adding a precommit[1] to start collecting data about the build times. It currently only mirrors the Java mvn install precommit. I'll gather data over the next two weeks and provide a summary here. You can rerun the precommit by issuing "Run Java Gradle

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-13 Thread Lukasz Cwik
There has been plenty of time for comments on the PR and the approach. So far Ken Knowles has provided the most feedback on the PR, Ken would you like to finish the review? On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau wrote: > This is only a setup thing and better to not break the mast

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-10 Thread Romain Manni-Bucau
This is only a setup thing and better to not break the master history for poc/tests, in particular when no very localized. Alternative can be to ask another temp repo to infra and have a synchro between both but dont think it does worth it personally. Le 10 nov. 2017 18:57, "Lukasz Cwik" a écri

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-10 Thread Lukasz Cwik
The reason to get it on master is because that is where all the PRs are. An upstream branch without any development means no data. Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins configuration on the branch because the seed job always runs against master. On Thu, Nov 9, 2017 at

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
What about pushing it on a "upstream" branch and testing it for 1 week in parallel of the maven reference build? If gradle is always 50% faster on jenkins then it could become master setup without much discussion I guess. We can even have 2 jenkins jobs: one with the daemon etc and one without. Al

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Lukasz Cwik
I wouldn't mind merging this change in so I could setup those Gradle Jenkins precommits. As per our contribution guidelines, any committer willing to sign off on the PR? On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau wrote: > Le 9 nov. 2017 21:31, "Kenneth Knowles" a écrit : > > Keep in mi

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
Le 9 nov. 2017 21:31, "Kenneth Knowles" a écrit : Keep in mind that a clean build is unusual during development (it is common for mvn use and that is a bug) and also not necessary for precommits if the build tool is correct enough that caching is safe. So while this number matters, it is not the

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Kenneth Knowles
Keep in mind that a clean build is unusual during development (it is common for mvn use and that is a bug) and also not necessary for precommits if the build tool is correct enough that caching is safe. So while this number matters, it is not the most important. On Thu, Nov 9, 2017 at 11:30 AM, Ro

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
I will try next week yes but the 2 runs i did were 28mn vs 32mn from memory - after having downloaded all deps once. Le 9 nov. 2017 19:45, "Lukasz Cwik" a écrit : > If Gradle was slow, do you mind running the build with --profile and > sharing that and also sharing the Maven build log? > > On Th

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Lukasz Cwik
If Gradle was slow, do you mind running the build with --profile and sharing that and also sharing the Maven build log? On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik wrote: > Romain, I don't understand your last comment, were you trying to say that > you had the same Gradle build times like I did

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Lukasz Cwik
Romain, I don't understand your last comment, were you trying to say that you had the same Gradle build times like I did and it was an improvement over Maven or that you did not and you experienced build times that were equivalent to Maven? On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau wrote

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
2017-11-09 18:38 GMT+01:00 Kenneth Knowles : > On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau > wrote: > >> (this is another topic so we can maybe open another thread) issue is >> not much about python but more about the fact the build is not self >> contained. it is a maven build and maven sh

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Kenneth Knowles
On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau wrote: > (this is another topic so we can maybe open another thread) issue is > not much about python but more about the fact the build is not self > contained. it is a maven build and maven should be sufficient without > having to install python

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
(this is another topic so we can maybe open another thread) issue is not much about python but more about the fact the build is not self contained. it is a maven build and maven should be sufficient without having to install python + dependencies. I don't see any technical blockers to do it (except

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Lukasz Cwik
Hmm, I have had good luck when following the Python quick start setup on multiple machines by ensuring the installed version of setuptools, virtualenv and pip are new enough versions. You can always skip the Python portion of the build by exclud

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
The 1.3.5 file is when i installed the python dependencies manually to make the build passing (the pip command never passed on my computer and therefore the build always has been broken until i installed it manually - independently from the build tool). Romain Manni-Bucau @rmannibucau | Blog | Ol

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Lukasz Cwik
It turns out that the Apache Rat Ant task and the Apache Rat Maven plugin differ in that the plugin automatically excludes certain files by default while the Ant task does not. See: http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.html#useDefaultExcludes I fixed the list to exclude ".ide

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-09 Thread Romain Manni-Bucau
Ok, the rat issues I got were: == File: /home/rmannibucau/1_dev/beam/.idea/* == File: /home/rmannibucau/1_dev/beam/sdks/python/=1.3.5 The first one could be in my default exclude - even if eclipse/idea files should be in the default exclude set of beam rat config IMHO, the last one is more a "?"

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Jean-Baptiste Onofré
Thanks for the update. I was swamped on some meetings. I'm back to test the latest changes. Regards JB On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik wrote: >Thanks everyone for trying this build out in different workspaces / >configurations. This will help make sure the build works for more >peo

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Lukasz Cwik
Thanks everyone for trying this build out in different workspaces / configurations. This will help make sure the build works for more people and will get rid of any rough edges. Performance (All): Maven performs parallelization at the module level, an entire module needs to complete before any dep

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Jean-Baptiste Onofré
Same for me for rat and python build too: FAILURE: Build completed with 2 failures. 1: Task failed with an exception. --- * What went wrong: Execution failed for task ':rat'. > Found 905 files with unapproved/unknown licenses. See file:/home/jbonofre/Workspace/beam/build/reports/rat/rat

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Romain Manni-Bucau
gradle branch doesnt build for me (some rat issues) Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-08 5:41 GMT+01:00 Jean-Baptiste Onofré : > Great ! > > What explain these difference ? I'm curious especially for the clean build > all Java modules: is it a questio

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-08 Thread Jean-Baptiste Onofré
Great ! What explain these difference ? I'm curious especially for the clean build all Java modules: is it a question of parallel execution ? Regards JB On 11/08/2017 02:59 AM, Lukasz Cwik wrote: The Gradle POC has made significant advances since last week (shading, Python, Go, Docker builds

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Ted Yu
Please see https://dzone.com/articles/java-command-line-interfaces-part-30-finale-observations On Tue, Nov 7, 2017 at 6:31 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Wow, these are impressive differences! Do you know why is that so? In > particular I'm surprised at the differ

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Ted Yu
Nice work. From my recent experience, gradle 4.3 is needed when building with java 9.0.1 See INFRA-15448 Original message From: Lukasz Cwik Date: 11/7/17 5:59 PM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [DISCUSS] Move away from Apache Maven as build tool The Gradle

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Eugene Kirpichov
Wow, these are impressive differences! Do you know why is that so? In particular I'm surprised at the difference in performance of the Java clean build: feels like the performed work ought to be the same. Does it all come from better parallelization? (in Maven I believe we don't specify "-T 1C" on

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-07 Thread Lukasz Cwik
The Gradle POC has made significant advances since last week (shading, Python, Go, Docker builds, ...). I believe the current state is close enough to the Maven build system to warrant a comparison. The largest build differences I noticed are: * Full build takes about ~22mins using Gradle (paralle

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Jean-Baptiste Onofré
That makes sense. The point is that we have to compare equivalently. I'm also curious about Gradle PoC assuming it does the same actions as Maven. Regards JB On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles wrote: >I'm confident that any choice will speed things up dramatically even >beyond >a

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Kenneth Knowles
I'm confident that any choice will speed things up dramatically even beyond a fast profile, even if the new tool runs all the extra stuff. But that is a question that we can answer empirically anyhow. Let's see how it goes! Incidentally, my experiments with Bazel have led me to the conclusion that

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Jean-Baptiste Onofré
Hi It's what I said in a previous e-mail: I don't think that just changing the build tool will improve a lot the build time. We already know (and discussed while ago) that plugins like findbugs, checkstyle, etc are taking time. So, I think we can already have a fast profile. Regards JB On No

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-03 Thread Romain Manni-Bucau
Hi guys, when you check the duration of each mojo of the build (almost since python part of the build just breaks it locally) you see that there is no real link with maven for the perf issues beam can encounter: https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9 (generated from t

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Kenneth Knowles
I have started one, here: https://github.com/kennknowles/beam/commits/bazel. It is not nearly as far along as Luke's. For the POC I am just putting things in one root BUILD, and learning where we might find the necessary plugins as I go. I am happy to grant push access to this branch. It would be s

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Ahmet Altay
Has anyone started a POC with Bazel? I would be interested in helping that effort. On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik wrote: > I have started a POC for using Gradle here: > https://github.com/lukecwik/incubator-beam/tree/gradle > > Things that work: > * compiling all Java code (src/main

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Lukasz Cwik
I have started a POC for using Gradle here: https://github.com/lukecwik/incubator-beam/tree/gradle Things that work: * compiling all Java code (src/main and src/test) * generating source from protos * generating source from avro * running rat, checkstyle Partially working: * generating maven pom

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Jean-Baptiste Onofré
Agree to move forward on a PoC. Thanks Reuven for bringing discussion on the mailing list ! Regards JB On Nov 1, 2017, 03:20, at 03:20, Reuven Lax wrote: >Some good discussion here, and thanks to JB and Romain for adding to >it! > >JB makes the good point that we still need to release Maven art

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Reuven Lax
Some good discussion here, and thanks to JB and Romain for adding to it! JB makes the good point that we still need to release Maven artifacts, as many Beam users want to develop using Maven. So none of this discussion will affect our release process, as we still need Maven "releases." At this po

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Charles Chen
As a contributor to the Beam Python SDK, I noticed that many of the points above regarding Maven and Gradle pertain mostly to Java SDK development. For Python development, Maven is much less natural, and we end up just shelling out to perform builds and tests. For Python SDK (and upcoming Go SDK d

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Robert Bradshaw
+1, Maven is both a build tool and a repository, and the latter is essential to keep. Both Gradel and Bazel can interface with this repository. I am, however, very supportive of moving away from Maven to a tool that supports correct incremental, hermetic, dependency-driven, multi-langauge, and hop

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Kenneth Knowles
Echoing what JB and Reuven said, we absolutely must provide maven central artifacts for Java users, just as we provide pypi artifacts for Python users. I see Maven as still a viable tool for single-module Java builds, especially considering its rich plugin ecosystem. On Mon, Oct 30, 2017 at 11:27

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-31 Thread Romain Manni-Bucau
Hmm, so the incremenal support is mainly in "shared" libs - which means not final plugins - but this is a work to do by plugin and not doable by maven core which can't do it by itself correctly - so can be worth opening an issue on plugins. Did you check gradle findbugs support was better and stil

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Reuven Lax
I think that's a very good point. No matter what build system we use for our own personal development, we still need to release Maven artifacts and releases as we need to support our users using Maven. On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré wrote: > Generally speaking, it's inter

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Jean-Baptiste Onofré
Generally speaking, it's interesting to evaluate alternatives, especially Gradle. My point is also to keep Maven artifacts and "releases" as most of our users will use Maven. For incremental build, afair, there's some enhancements on Maven but I have to take a look. Regards JB On Oct 31, 2017,

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Eugene Kirpichov
Hi! Many of these points sound valid, but AFAICT Maven doesn't really do incremental builds [1]. The best it can do is, it seems, recompile only changed files, but Java compilation is a tiny part of the overall build. Almost all time is taken by other plugins, such as unit testing or findbugs - a

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Romain Manni-Bucau
Hi Even if not a commiter or even PMC, I'd like to mention a few points from an external eye: - Maven stays the most common build tool and easier one for any user. It means it is the best one to hope contributions IMHO. - Maven has incremental support but if there is any blocker the community is

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Jean-Baptiste Onofré
-0 For the following reasons reasons: - maven is a Apache project and we can have support/improvement - I don't see how another build tool would speed up the build by itself - Apache default release process is based on Maven On the other hand, Gradle could be interesting. Anyway it's something to

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Chamikara Jayalath
+1 for exploring other build tools to improve the build time and for better cross language support. But, as others mentioned, we should continue to support Maven based builds for some time till things are fully migrated. For example, we have updated PerfKitBenchmarker to execute Beam jobs through M

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Reuven Lax
One more comment: regardless of which build tool we use for development, we still need to publish Maven artifacts to support Maven users of Beam. On Mon, Oct 30, 2017 at 10:46 AM, Ted Yu wrote: > I agree with Ben's comment. > > Recently I have been using gradle in another Apache project and foun

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Ted Yu
I agree with Ben's comment. Recently I have been using gradle in another Apache project and found it interesting. Cheers

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Reuven Lax
I wonder if we could prototype both Bazel and Gradle to do a better comparison (and also to compare the results with our current Maven build). On Mon, Oct 30, 2017 at 10:32 AM, Ben Chambers wrote: > I think both Gradle and Bazel are worth exploring. Gradle is definitely > more common in the wild

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Ben Chambers
I think both Gradle and Bazel are worth exploring. Gradle is definitely more common in the wild, but Bazel may be a better fit for the large mixture of languages being developed in one codebase within Beam. It might be helpful for us to list what functionality we want from such a tool, and then hav

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Reuven Lax
I will add one more thing I think we all want from a build tool - good IDE integration with IntelliJ and Eclipse. On Mon, Oct 30, 2017 at 10:27 AM, Kenneth Knowles wrote: > I also support exploring a move away from Apache Maven for orchestrating > our build. > > For a single-module project, I st

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Henning Rohde
+1 to the initiative. It would great to have better support for Go and Docker container images. The current Go maven integration in particular is clunky [1], but I'll have to look into the details of the alternatives to see if they are better. Henning [1] https://github.com/apache/beam/blob/maste

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Kenneth Knowles
I also support exploring a move away from Apache Maven for orchestrating our build. For a single-module project, I still think it can be a good build tool, and we could still use it for this sort of thing, but I think we are reaching a multi-module scale where it does not work well. Almost all of

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Reuven Lax
If we're looking at other build options, I think we should also consider Bazel. Some advantages I see of Bazel over Gradle: Gradle is reportedly much slower than Bazel. I haven't measured this myself, but there are many user reports of slow builds, memory leaks, etc. Bazel's build language is bas

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Eugene Kirpichov
This seems like a very good idea. With the effective complete lack of incremental builds in Maven, it's frustrating to routinely spend several minutes re-verifying a PR after fixing a checkstyle warning in an extension module. Another non-Apache alternative could be Bazel, which is even faster and

[DISCUSS] Move away from Apache Maven as build tool

2017-10-30 Thread Lukasz Cwik
I wanted to make this thread more visible. This discussion stems from Ken's thread about Jenkins pre/post commit issues[1]. I did some investigation as for ways to improve the quality of the signal from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I had limited success but eve