2018-01-15 11:54 GMT+01:00 Yoann Rodiere <yo...@hibernate.org>: > > We should reconfigure those to not "install" - that's actually a bad > > habit, legacy from Maven 2 times - people nowadays recommend using > > "mvn clean verify", especially on CI environments. > > I could not agree more, that would be cleaner, but that's not possible. And > believe me, I tried hard. Last time I checked, some of the plugins we use > with dynamic dependency resolution would ignore the artifacts being built, > and would always fetch the artifacts from the Maven repos (for SNAPSHOTs, > they would end up using nightlies). > I'm not talking about when we use standard maven markup to declare > dependencies, but when the plugin itself has to fetch dependencies > "dynamically", which happens when we setup a WildFly server with our own > modules in particular. See maven-dependency-plugin's "artifactItems" > configuration. >
Yes, I wanted to bring this up, too. I believe it's a similar issue for the OSGi-based tests. The brute-force solution is to actually work with "install", but have a job-local Maven repo to prevent any side-effects across jobs. That won't be great for build times, though, unless perhaps a local Nexus would be available as a cache on each slave. > On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero <sa...@hibernate.org> wrote: > > > On 15 January 2018 at 08:42, Yoann Rodiere <yo...@hibernate.org> wrote: > > > Thanks Sanne ! > > > > > > I have one question... > > > > > >> Please never rely on this as "storage": it's just meant as cache and > > >> we reserve the right to wipe it all out at any time. > > > > > > I gather you say that so that we don't try to "release" artifacts into > > this > > > cache? But temporary storage for the duration of one build will still > be > > > safe? > > > > > > Because our builds obviously rely on the local repository for > short-term > > > storage (for the duration of the build). For example the dependencies > are > > > only checked and downloaded if necessary at the beginning of the build, > > and > > > then are expected to exist in the local repository until the build > stops. > > > Another example: our WildFly modules are first built and installed in > the > > > "modules" subproject, and later "fetched" from the local repository in > > the > > > "integrationtest/wildfly" subproject. > > > > > > If we were to clear the cache during a build, things would probably go > > > wrong. Worse, if two parallel builds were to install the same artifacts > > > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run the > > risk > > > of testing the wrong "version" of this artifact in one of the builds... > > > > SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs > > could conflict with the regular master jobs. > > We should reconfigure those to not "install" - that's actually a bad > > habit, legacy from Maven 2 times - people nowadays recommend using > > "mvn clean verify", especially on CI environments. > > > > I agree about the perils of clearing the cache during in-progress builds > > too. > > > > I just meant to warn that we don't have any backup plan in place, and > > I do plan to just wipe the whole thing occasionally: > > - when we have any direct need, e.g. currupted downloads > > - when it gets too large > > - if it gets too expensive > > - regularly, just to "practice" that everything works with an empty > cache > > > > Also our "disaster recovery" plan to rebuild all infrastructure will > > always assume it's ok to reboot with having this file system empty. > > > > Thanks, > > Sanne > > > > > > > > > > > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero <sa...@hibernate.org> > > wrote: > > >> > > >> Hi all, > > >> > > >> while the new build machines are fast, some of you pointed out we're > > >> now spending a relative high amount of time downloading maven > > >> dependencies, this problem being compounded by the fact we "nuke" idle > > >> slaves shortly after they become idle. > > >> > > >> I just spent the day testing a distributed file system, and it's now > > >> running in "production". > > >> It's used exclusively to store the Gradle and Maven caches. This is > > >> stateful and independent from the lifecycle of individual slave nodes. > > >> > > >> Unfortunately this solution is not viable for Docker images, so while > > >> I experimented with the idea I backed off from moving the docker > > >> storage graph to a similar device. Please don't waste time trying that > > >> w/o carefully reading the Docker documentation or talking with me :) > > >> Also, beyond correctness of storage semantics, it's likely far less > > >> efficient for Docker. > > >> > > >> To learn more about our new cache: > > >> - > > >> > > https://github.com/hibernate/ci.hibernate.org/commit/ > dc6e0a4bd09fb3ae6347081243b4fb796a219f90 > > >> - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html > > >> > > >> I'd add that - because of other IO tuning in place - writes might > > >> appear out of order to other nodes, and conflicts are not handled. > > >> Shouldn't be a problem since snapshots now have timestamps, but this > > >> might be something to keep in mind. > > >> > > >> N.B. > > >> Please never rely on this as "storage": it's just meant as cache and > > >> we reserve the right to wipe it all out at any time. > > >> > > >> Thanks, > > >> Sanne > > >> _______________________________________________ > > >> hibernate-dev mailing list > > >> hibernate-dev@lists.jboss.org > > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > > > > > > > > -- > > > Yoann Rodiere > > > yo...@hibernate.org / yrodi...@redhat.com > > > Software Engineer > > > Hibernate NoORM team > > > > > -- > Yoann Rodiere > yo...@hibernate.org / yrodi...@redhat.com > Software Engineer > Hibernate NoORM team > _______________________________________________ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
