We currently use Pants to manage our huge python deployment and it's been a godsend for dependency management. We also have a bunch of clojure based kafka consumers (and a few APIs) that live in the same repo, but are built manually with lein and versioned/deployed separately. We've wanted a pants-like system for our clojure builds for a while now. I looked at building a plugin for pants (as have others https://groups.google.com/forum/#!topic/pants-devel/PPIAEKzKm8w) as Pants does support java, however it's based on incremental builds with nailgun and seems to build a separate jar per directory (fitting in with the pants philosophy). I'm not sure I like this model, and there doesn't seem to be an nailgun-like solution for clojure anyway.
We're starting to lean towards gradle as a top-level build system that would allow us to declare internal dependencies and keep 3rd party artifact versions in sync. I think this is the cleanest solution. Gradle's very flexible and it's easy to add new functionality to it. On Thursday, June 7, 2018 at 1:03:45 AM UTC-4, Reid McKenzie wrote: > > I previously worked with pants [1] in the context of an extremely large > (~50GiB) repository which saw a huge amount of daily churn. In that context > build tools like pants and Bazel become absolutely essential because the > only way to have a reasonable CI/CD system and build/test times is to do > change analysis to predict what tests and other products are impacted. > While I think that Pants, Buck [2], Bazel and other Google Blaze > derivatives have made fundamental and perhaps undesirable complexity > sacrifices in order to be polygot, they do add significant value to a large > organization. > > Presently, I work on a much smaller Clojure project encompassing several > microservices which communicate over Kafka. Each service is its own > sub-project in a single repository. We coordinate modules using > lein-modules [3]. For a variety of reasons, the majority of our interesting > tests involve Kafka and potentially integrating against other services > continually built out of the same repo. These tests are wall clock > expensive due to a variety of system initialization overheads which are > difficult to amortize. > > It has already proven very profitable for us to deploy a limited amount of > test invalidation by using the inter-module dependency information > lein-modules exposes to restrict test runs to those modules which have seen > source changes or which according to topological sort order may depend on > impacted modules. This is a correct but non-minimal heuristic which leaves > time on the table. Our deployment artifacts are also fairly expensive to > produce, and so I'm very interested in the possibility of a polygot tool > which can capture dependency information both for test invalidation and > build product invalidation. > > It is in theory possible to write a "macro" (function) for either Pants or > Buck which defines a clojure_library or clojure_app target in terms of > simply generating a java_library which happens to have your clj(c) files > for resources and depend on Clojure. I got this working for Buck at one > point, but Buck doesn't support Maven dependencies and requires that you > hand-manage all your jars which made the ergonomics unfortunate. I haven't > taken a serious crack at getting Pants wired up that way. I worked with the > pants developers at one point to understand what they thought it would take > to get a clojure_library target integrated "the right way" and it turns out > to be a mess because of how pants' JVM model is coupled to Ivy, Scala's > incremental compiler. > > I've been sketching at Katamari [4] on and off, the thought being that the > fundamental dependency and invalidation concerns are well enough understood > [7] that maybe by trying to build a framework or fire customers it's > possible to achieve a meaningful complexity reduction. I haven't managed > that yet. Of particular frustration is notions of when to build classpaths > and resolve dependencies - although forcing dependency pinning fixes this > which is Buck's answer. > > There's also Juxt's mach [5] (not to be confused with Mozilla's build tool > of the same name), krei.alpha [6] which has some of the same goals as > katamari and I've seen some sketches at multi-module boot. > > In short, I do think that more sophisticated build tools make sense in > the Clojure space. Sure Clojure itself doesn't have traditional compilation > needs in the sense of say C++ or Scala, but notions of products and static > dependency information is more generally valuable and can help with whole > applications or systems. After spending a fair bit of time evaluating the > options, I concur with Colin. If lein-modules or lein-monolith doesn't > solve your problem Gradle is probably the most mature "off the shelf" > solution. > > Reid > > [1] https://pantsbuild.org/ <https://www.pantsbuild.org/> > [2] https://buckbuild.com/ > [3] https://github.com/FundingCircle/lein-modules > [4] https://github.com/arrdem/katamari > [5] https://github.com/juxt/mach > [6] https://github.com/SevereOverfl0w/krei.alpha > [7] > https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf > > On Mon, Jun 4, 2018, at 10:04 PM, Nathan Fisher wrote: > > It's not something that as a user you'd be exposed to. The easiest way to > demonstrate the impact is to write a library and test target in two > different ways for a Java project. > > 1. using java_library target for the library. > 2. using genrule for the library that calls out to javac. > > Run the test target, introduce whitespace into the library, and run the > test target again. It should only do a full execution in the 2nd example. > > On Sunday, 15 April 2018 17:09:12 UTC-3, Gregg Reynolds wrote: > > > > On Mon, Jan 29, 2018, 2:07 PM Nathan Fisher <nfi...@junctionbox.ca> wrote: > > > ... > > In order to get the speed that the build tool promises you need the > ability to efficiently parse the ABI of each file and only build its > dependents when the ABI has changed. > > > Can you please elaborate on this a bit? I've been using Bazel for a > project that involve C and Java, with cross-compiles, and ABI parsing has > never come up. It just works. > > > > I’m not certain you can do that with Clojures compiler but someone else > might be able to chime in with a way that you could. > > > I like boot for clojure stuff, not sure Bazel offers any advantage. > Clojurescript may be a different story, tho. > > So far I cannot see a good reason to switch from boot to Bazel for 100% > clojure projects. Mixed language - clojure, clojurescript, js, Java, > kotlin, JNI,etc - may be a different story, still not sure. > > Then there's remote caching: > https://docs.bazel.build/versions/master/remote-caching.html. a hyuuuuge > win for many scenarios, not so sure about clojure. > > G > G > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clo...@googlegroups.com <javascript:> > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+u...@googlegroups.com <javascript:> > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+u...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
