Simplifying examples/java precommit is encompassed in BEAM-4033. Both you and Kenn pointed to the same thing. To my knowledge it is the only place where we have such a loop that generates tasks.
On Mon, Apr 9, 2018 at 3:37 PM Romain Manni-Bucau <[email protected]> wrote: > > > Le 9 avr. 2018 21:04, "Lukasz Cwik" <[email protected]> a écrit : > > Romain, can you clarify by the weird task naming? (Give some examples > using our current project and Gradle and what you would have expected.) > > > Sure > > > https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78 > for instance > > It is way more natural to have one generic command you can select a runner > (including a csv value) than N commands for this kind of things. > > Gradle allows to define custom tasks and "scopes". This must not be used > at Xtrem until justifiable. We now have a lot of configurations not making > much sense too until you scripted yourself the script or know beam well. > This doesnt enable contribs :(. > > Hope it is clearer > > > On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau <[email protected]> > wrote: > >> +1 gradle not being mainstream - and even more in beam scope than other >> java scope - it must stay simple. I know that personally it takes me x2 or >> 3 to say "ok ill fork" a project using gradle, in particular with custom >> scripts and not only parent/module scripts. >> >> If needed we can write custom plugins to simply it and hide the >> complexity and weird task naming can move to "standard" names with >> options/parameters maybe? >> >> Le 9 avr. 2018 20:27, "Kenneth Knowles" <[email protected]> a écrit : >> >>> My phrasing sounded a bit too much like adding a blocking condition. I >>> absolutely don't want to do that. Our maven build was far from clear, and >>> had lots of tech debt and spooky action at a distance, >>> >>> I just want to emphatically agree with JB's sentiment that we have an >>> opportunity to improve it before it ossifies in the current state. Maybe we >>> won't really have time, and that's OK too and we can just improve as we go. >>> >>> Luke - I know we have different standards of clarity but I doubt we'll >>> have specific disagreements on cleanups to the build as they happen.* I was >>> mostly listing elementary programming concepts that apply to our build and >>> tend to get forgotten in configs & build files. Being "natural gradle" is >>> also important. Let's see how it goes. >>> >>> Kenn >>> >>> *FWIW I did know that we have some string-munged identifiers already, >>> but mostly I am speaking from experience beyond & before Beam too. I'll >>> likely tidy the examples build.gradle as I finish up nexmark in a similar >>> way. >>> >>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik <[email protected]> wrote: >>> >>>> JB, learning the build system in a project is hopefully avoided by most >>>> users if the contribution guide can clearly explain what users need to do. >>>> But for everyone else who wants to change a dependency version or add a >>>> dependency it should be as simple as copy/paste (which I believe it already >>>> is). Note that people who want to do anything more complicated like add new >>>> jenkins job configurations for new integration test targets needs to learn >>>> how the build system works, how maven profiles work, how to plumb the >>>> required set of attributes from Jenkins into the test target via the build >>>> system. >>>> >>>> Kenn, I have to disagree with a lot in 1 and 2. For users who are >>>> unfamiliar with Maven, we didn't setup the Maven build in such a way that >>>> anyone unfamiliar with Maven knew what was going on or try to use concepts >>>> unnatural to Maven in an attempt to make it seem easier. I believe we >>>> should stick with Gradle/Groovy conventions. Users who are not familiar >>>> with how Gradle/Groovy works will either ask the community or look at >>>> StackOverflow and doing things like passing the project around into >>>> functions is extremely uncommon compared to using the current context and >>>> applying closures to it. For users who are familiar with Gradle, the builds >>>> should be natural Gradle. >>>> >>>> *- If you are going to refer to an identifier, it should have an >>>> explicit point of origin (not strings smashed together in a loop)* >>>> I assume your referring to how the examples java precommit is done. It >>>> is the only case of it to my knowledge and extra hands to migrate it to be >>>> like how the validates runner tests run would be useful. Filed >>>> https://issues.apache.org/jira/browse/BEAM-4033 >>>> >>>> >>>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles <[email protected]> wrote: >>>> >>>>> Huge +1 to the idea of investing in simplification and polish of the >>>>> Gradle files before considering the migration complete. >>>>> >>>>> 1. build.gradle files should be as close to straight-line >>>>> configuration as possible: >>>>> - You should be able to understand a module's build easily, locally, >>>>> without knowing Groovy >>>>> - As little Groovy programming as possible, focused on providing >>>>> well-defined utilities >>>>> >>>>> 2. Explicit > implicit, especially for config files >>>>> - Ditto about being able to understand a module's build locally >>>>> - Minimize globally inherited config, in favor of explicitly calling >>>>> utilities >>>>> - Passing current project to a utility is better than trying to make >>>>> something the "closure" >>>>> - If you are going to refer to an identifier, it should have an >>>>> explicit point of origin (not strings smashed together in a loop) >>>>> >>>>> When in doubt, a little redundancy to improve readability is worth it >>>>> in this context. >>>>> >>>>> Kenn >>>>> >>>>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I did multiple gradle build since last week and I would like to share >>>>>> one of my concern: it's about the communities. >>>>>> >>>>>> If I think our users won't see any change for them due to Gradle build >>>>>> (I think that most of our users will still use Maven with artifacts >>>>>> provided by Gradle), I'm more concerned by the dev community and the >>>>>> contribution. >>>>>> >>>>>> Maven is well known and straight forward for a large part of potential >>>>>> contributors. I think we have to keep in mind that we still have to >>>>>> grow >>>>>> up our contributors community. >>>>>> >>>>>> Today, maybe I'm wrong, but I have the feeling that gradle build is >>>>>> not >>>>>> straight forward (build.gradle includes build_rules.gradle, gathering >>>>>> all taks all together). >>>>>> >>>>>> I would like to add a task in the gradle "migration" process: simplify >>>>>> the gradle structure and files, and document this. >>>>>> >>>>>> I know we already have a Jira about the documentation part, but I >>>>>> would >>>>>> like to "polish" and use a clean structure for the Gradle resources. >>>>>> As >>>>>> already quickly discussed, I think that having one gradle file per >>>>>> tasks >>>>>> in the .gradle directory would be helpful. >>>>>> >>>>>> The goal is really to simplify the contribution. >>>>>> >>>>>> Do you agree if I add a Jira about "Gradle polish" ? >>>>>> Thoughts ? >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 07/04/2018 04:52, Scott Wegner wrote: >>>>>> > Here's an end-of-day update on migration work: >>>>>> > >>>>>> > * Snapshot unsigned dailies and signed release builds are working >>>>>> (!!). >>>>>> > PR/5048 [1] merges changes from Luke's branch >>>>>> > * python precommit failing... will investigate python precommit >>>>>> Monday >>>>>> > * All Precommits are gradle only >>>>>> > * All Postcommits except performance tests and >>>>>> Java_JDK_Versions_Test >>>>>> > use gradle (after PR/5047 [2] merged) >>>>>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to >>>>>> be >>>>>> > merged before switching >>>>>> > * ValidatesRunner_Spark failing consistently; investigating >>>>>> > >>>>>> > Thanks for another productive day of hacking. I'll pick up again on >>>>>> Monday. >>>>>> > >>>>>> > [1] https://github.com/apache/beam/pull/5048 >>>>>> > [2] https://github.com/apache/beam/pull/5047 >>>>>> > >>>>>> > >>>>>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau >>>>>> > <[email protected] <mailto:[email protected]>> wrote: >>>>>> > >>>>>> > Why building a zip per runner which its stack and just pointing >>>>>> out >>>>>> > on that zip and let beam lazy load the runner: >>>>>> > >>>>>> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... >>>>>> (or >>>>>> > the fromSystemProperties() if it gets merged a day ;)) >>>>>> > >>>>>> > Le 6 avr. 2018 20:21, "Kenneth Knowles" <[email protected] >>>>>> > <mailto:[email protected]>> a écrit : >>>>>> > >>>>>> > I'm working on finding a solution for launching the Nexmark >>>>>> > suite with each runner. This doesn't have to be done via >>>>>> Gradle, >>>>>> > but we anyhow need built artifacts that don't require user >>>>>> > classpath intervention. >>>>>> > >>>>>> > It looks to me like the examples are also missing this - >>>>>> they >>>>>> > have separate configuration e.g. sparkRunnerPreCommit but >>>>>> that >>>>>> > is overspecified compared to a free-form launching of a >>>>>> main() >>>>>> > program with a runner profile. >>>>>> > >>>>>> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik < >>>>>> [email protected] >>>>>> > <mailto:[email protected]>> wrote: >>>>>> > >>>>>> > Romain, are you talking about the profiles that exist as >>>>>> > part of the archetype examples? >>>>>> > >>>>>> > If so, then those still exist and haven't been changed. >>>>>> If >>>>>> > not, can you provide a link to the profile in a pom >>>>>> file to >>>>>> > be clearer? >>>>>> > >>>>>> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau >>>>>> > <[email protected] <mailto:[email protected]>> >>>>>> wrote: >>>>>> > >>>>>> > Hi Scott, >>>>>> > >>>>>> > is it right that 2 doesn't handle the hierachy >>>>>> anymore >>>>>> > and that it doesn't handle profiles for runners as >>>>>> it is >>>>>> > currently with maven? >>>>>> > >>>>>> > >>>>>> > Romain Manni-Bucau >>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | >>>>>> Blog >>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> > <http://rmannibucau.wordpress.com> | Github >>>>>> > <https://github.com/rmannibucau> | LinkedIn >>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book >>>>>> > < >>>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance >>>>>> > >>>>>> > >>>>>> > 2018-04-06 18:32 GMT+02:00 Scott Wegner >>>>>> > <[email protected] <mailto:[email protected]>>: >>>>>> > >>>>>> > I wanted to start a thread to summarize the >>>>>> current >>>>>> > state of Gradle migration. We've made lots of >>>>>> good >>>>>> > progress so far this week. Here's the status >>>>>> from >>>>>> > what I can tell-- please add or correct >>>>>> anything I >>>>>> > missed: >>>>>> > >>>>>> > * Release artifacts can be built and published >>>>>> for >>>>>> > Snapshot and officlal releases [1] >>>>>> > * Gradle-generated releases have been validated >>>>>> with >>>>>> > the the Apache Beam archetype generation >>>>>> quickstart; >>>>>> > still needs additional validation. >>>>>> > * Generated release pom files have correct >>>>>> project >>>>>> > metadata [2] >>>>>> > * The python pre-commits are now working in >>>>>> Gradle [3] >>>>>> > * Ismaël has started a collaborative doc of >>>>>> Gradle >>>>>> > tips [4] as we all learn the new system-- >>>>>> please add >>>>>> > your own. This will eventually feed into >>>>>> official >>>>>> > documentation on the website. >>>>>> > * Łukasz Gajowy is working on migrating >>>>>> performance >>>>>> > testing framework [5] >>>>>> > * Daniel is working on updating documentation to >>>>>> > refer to Gradle instead of maven >>>>>> > >>>>>> > If I missed anything, please add it to this >>>>>> thread. >>>>>> > >>>>>> > The general roadmap we're working towards is: >>>>>> > (a) Publish release artifacts with Gradle >>>>>> (SNAPSHOT >>>>>> > and signed releases) >>>>>> > (b) Postcommits migrated to Gradle >>>>>> > (c) Migrate documentation from maven to Gradle >>>>>> > (d) Migrate perfkit suites to use Gradle >>>>>> > >>>>>> > For those of you that are hacking: thanks for >>>>>> your >>>>>> > help so far! Progress is being roughly tracked >>>>>> on >>>>>> > the Kanban [6]; please make sure the issues >>>>>> assigned >>>>>> > to you are up-to-date. Many of the changes are >>>>>> > staged on lukecwik's local branch [7]; we'll >>>>>> work on >>>>>> > merging them back soon. >>>>>> > >>>>>> > >>>>>> > [1] >>>>>> https://github.com/lukecwik/incubator-beam/pull/7 >>>>>> > [2] >>>>>> https://github.com/lukecwik/incubator-beam/pull/3 >>>>>> > [3] https://github.com/apache/beam/pull/5032 >>>>>> > [4] >>>>>> > >>>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit >>>>>> > [5] https://github.com/apache/beam/pull/5003 >>>>>> > [6] >>>>>> > >>>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242 >>>>>> > >>>>>> > [7] >>>>>> > >>>>>> https://github.com/lukecwik/incubator-beam/tree/gradle >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > Got feedback? http://go/swegner-feedback >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > Got feedback? http://go/swegner-feedback >>>>>> >>>>> >
