Re: [hibernate-dev] ORM Jenkins Builds
On 27 March 2015 at 22:25, Steve Ebersole wrote: > We should remove that -XX:MaxPermSize=512m setting as we discussed before. I've changed this to: -server -Xmx2G -Xms800M -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -Xloggc:/home/jenkins/gradle-gcdetails.log -XX:+PrintFlagsFinal -XX:NewRatio=1 -XX:+UseG1GC The logging option should not slow down significantly, but let us know if there ever are issues with heap memory. > I understand all of that. I still think its probably worthwhile to look at > minimizing the amount of time and resources. For a couple of reasons. > First I am doing a lot of dev right now. In the 40-50 minutes it takes to > run the CI job after a push I often have other pushes queued up. Then when > that first jobs fails, I need to go back and shift gears back into that > first issue again. And spending roughly 75-80% of that time in a task that > we don't even fail the build for seems excessive. > > My preliminary thinking here is to define a more than one job. Setting > aside the idea of db-jobs for the moment... > > 1) Per-push job - triggered by push > a) runs tests > b) simplified checks (findbugs, checkstyle) that we actually fail the build > for > c) publish > > 2) Nightly build - triggered nightly (only with change in source?) > a) full checks (both failures and warnings) > b) aggregated javadocs > > * I'd actually like the nightly build to trigger once a day when a push > actually occured. Not sure that is possible. +1, feel free to create as many jobs as needed. There is a "clone job" option which I've found handy. > The "per-push" job really is a "main" one meant to run against H2. We still > need to set up the db specific jobs: Right, I even suspect that using the locally installed MySQL or PostgreSQL instances could result in faster builds than with H2 > > 3) DB job - triggered on successful execution of (1)? > a) run tests > > And ultimately I'd love to setup some jobs for JDK testing that pull the > artifacts built by (1) and tests them in a different JDK. Currently we need > the Java 8 JDK to do a full build of ORM (JSR 310 support). For the modules > other than hibernate-java8 we set source and target compatibility to Java 6. > We could simply set up some jobs that use Java 6 JDK for the build, but > that's not quite the same as pulling the artifacts built using Java 8 and > targetting Java 6 and trying to run those in Java 6 or 7 environment. I > don't know if Gradle has any existing support for this or whether this is > another plugin I will have to develop to make that happen. Consider also that since we now have N slaves, there is no guarantee that a second job will run on the same build slave. So if the JDK8 build produces artifacts but only deployes them in the local repository, the second node might not find the same as it has an independent repository cache. One way would be to actually upload the snapshots to the jboss Nexus, but it could be easier / more efficient to have a single job which does the compilation & packaging only with JDK8 but then runs the tests with JDK6 ? The JDK paths are well defined, so that should be scriptable with a couple of sequential invocations to Gradle? Sanne > > > On Fri, Mar 27, 2015 at 4:38 PM, Sanne Grinovero > wrote: >> >> Hi Steve, >> don't worry for the resources they consume. Each build job which we >> run is fully isolated and running on independent hardware, so it's not >> going to slow down any other build. >> >> Since we now have a set of small servers, one each dedicated for a job >> at a time, I've opted for low spec machines as an experiment. This >> performance is however disappointing, we can try different types of >> CPU/Memory/Disk categories. >> >> The current specs of each build node are: >> 2 CPUs >> 3 GB of RAM >> 18GB of root disk (of which 9GB available for our builds) >> 50GB available to run various docker images and temporary databases >> (allocated per job need) >> >> They run Fedora 21, Cloud Edition. >> >> As you can see, resources on a single build node are very limited - >> especially space and memory. >> But since there is no swap space configured (!) I expect that in case >> the memory is not enough, it will crash (and therefore let us know) >> rather than slow down. >> It's possible though that since the ORM builds allocate 2GB of heap, >> we're preventing the operating system from doing proper filesystem >> caching. >> >> currently we have: >> GRADLE_OPTS="-Xmx2G -Xms2G -XX:MaxPermSize=512m" >> >> I'm going to try to lower that, bear with me if there are failures. If >> that doesn't work we can definitely replace these with larger >> instances, it should take us just some minutes as it's all automated: >> the only reason for me to be conservative is to make fair use of these >> (free) resource, and also I'd like us to actually test for not needing >> much ram - we can get there in smaller steps as nee
Re: [hibernate-dev] ORM Jenkins Builds
On Sat, Mar 28, 2015 at 11:26 AM, Sanne Grinovero wrote: > > Consider also that since we now have N slaves, there is no guarantee > that a second job will run on the same build slave. > So if the JDK8 build produces artifacts but only deployes them in the > local repository, the second node might not find the same as it has an > independent repository cache. > This is one of the major differences between Gradle and Maven. Gradle has no idea about a "local repository" in terms of publishing. These jobs, if they publish, always publish to Nexus. We added the ability to read artifacts from "Maven local" mainly to deal with cases where we have local Maven-built projects that we want to have ORM pick up a local snapshot of. HCANN was the main thing here. But the "Gradle way" is to have the "other" local project (e.g., HCANN) publish its SNAPSHOT to Nexus and for the ORM build to pick it up from there. More repeatable builds that way. You pointed out one such scenario. Of course locally, Gradle (with some help usually) also lets you aggregate these disparate projects together into one cohesive build. And given that IIRC I converted HCANN to use Gradle for building as well, maybe its time to pull that ability. One way would be to actually upload the snapshots to the jboss Nexus, > but it could be easier / more efficient to have a single job which > does the compilation & packaging only with JDK8 but then runs the > tests with JDK6 ? > The JDK paths are well defined, so that should be scriptable with a > couple of sequential invocations to Gradle? That is another option for sure. And actually it does not even have to be separate invocations. We could script the creation of a new Test task based on the configured JDK6 path and just have both run in the same Gradle build. ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] ORM Jenkins Builds
Gradle really is quite elegant in most cases:
if ( project.name != 'hibernate-java8' && rootProject.hasProperty(
"JDK6_HOME" ) ) {
final File java6Home = new File( rootProject.property( "JDK6_HOME" ) as
String );
final org.gradle.internal.jvm.Jvm java6Jdk =
org.gradle.internal.jvm.Jvm.forHome( java6Home ) as
org.gradle.internal.jvm.Jvm;
if ( !java6Jdk.javaVersion.java6 ) {
throw new GradleException( "JDK6_HOME (if set) must point to a Java 1.6
home, found ${java6Jdk.javaVersion}" );
}
final Test mainTestTask = subProject.tasks.test as Test;
task testWithJdk6(type:Test) {
inputs = mainTestTask.inputs
outputs = mainTestTask.outputs
if ( java6Jdk.toolsJar ) {
bootstrapClasspath java6Jdk.toolsJar
}
classpath = java6Jdk.runtimeJar + mainTestTask.classpath
executable = java6Jdk.javaExecutable
// should we recompile the tests using JDK6? If so, set a different
classes dir
// here and clone new JavaCompile task for tests. But just running with
the Java 6
// JRE should be enough
testClassesDir = mainTestTask.testClassesDir
testSrcDirs = mainTestTask.testSrcDirs
enableAssertions = mainTestTask.enableAssertions
environment = mainTestTask.environment
systemProperties = mainTestTask.systemProperties
scanForTestClasses = mainTestTask.scanForTestClasses
includes = mainTestTask.includes
excludes = mainTestTask.excludes
jvmArgs = mainTestTask.jvmArgs
maxHeapSize = mainTestTask.maxHeapSize
minHeapSize = mainTestTask.minHeapSize
}
}
On Sat, Mar 28, 2015 at 11:46 AM, Steve Ebersole
wrote:
> On Sat, Mar 28, 2015 at 11:26 AM, Sanne Grinovero
> wrote:
>>
>> Consider also that since we now have N slaves, there is no guarantee
>> that a second job will run on the same build slave.
>> So if the JDK8 build produces artifacts but only deployes them in the
>> local repository, the second node might not find the same as it has an
>> independent repository cache.
>>
>
> This is one of the major differences between Gradle and Maven. Gradle has
> no idea about a "local repository" in terms of publishing. These jobs, if
> they publish, always publish to Nexus.
>
> We added the ability to read artifacts from "Maven local" mainly to deal
> with cases where we have local Maven-built projects that we want to have
> ORM pick up a local snapshot of. HCANN was the main thing here. But the
> "Gradle way" is to have the "other" local project (e.g., HCANN) publish its
> SNAPSHOT to Nexus and for the ORM build to pick it up from there.
>
> More repeatable builds that way. You pointed out one such scenario. Of
> course locally, Gradle (with some help usually) also lets you aggregate
> these disparate projects together into one cohesive build.
>
> And given that IIRC I converted HCANN to use Gradle for building as well,
> maybe its time to pull that ability.
>
>
> One way would be to actually upload the snapshots to the jboss Nexus,
>> but it could be easier / more efficient to have a single job which
>> does the compilation & packaging only with JDK8 but then runs the
>> tests with JDK6 ?
>> The JDK paths are well defined, so that should be scriptable with a
>> couple of sequential invocations to Gradle?
>
>
> That is another option for sure. And actually it does not even have to be
> separate invocations. We could script the creation of a new Test task
> based on the configured JDK6 path and just have both run in the same Gradle
> build.
>
>
___
hibernate-dev mailing list
hibernate-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
