Hi Chris,
Thanks a lot for looking into it. From the CXF perspective, I have seen that
many CXF builds have been aborted
because of the connection with master is lost (don't have exact builds to point
since we keep only last 3),
that could probably explain the hanging builds.
Best Regards,
Andriy Redko
CL> Another incident of CXF build junk sticking around on H40, although in that
case, the machine appears to be hosed
CL> because of broken container jobs from 2019, with over 1100(!) processes
identical to:
CL> containerd-shim -namespace moby -workdir
CL>
/var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc
CL> -address /run/containerd/containerd.sock -containerd-binary
/usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
CL> /sigh
CL> asf999@asf940:~$ ps wwaxu | grep CXF
CL> jenkins 8380 0.0 0.5 39795216 541148 ? Sl Jan22 5:06
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp
surefire_355731680319470622184tmp
CL> jenkins 16834 0.0 0.5 39733804 496308 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp
surefire_352090757718230945819tmp
CL> jenkins 25793 0.0 0.4 39797272 471988 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp
surefire_351240114022660372795tmp
CL> jenkins 28293 0.0 0.5 39751220 521156 ? Sl Jan21 6:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp
surefire_35221823887365033180tmp
CL> jenkins 28431 0.1 1.2 6242916 1218032 ? Sl Jan23 6:35
/home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath
/home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
-Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL>
-Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL>
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository
-U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> jenkins 36663 0.0 0.5 39697712 556556 ? Sl Jan22 5:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar
CL>
/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp
surefire_34483670662819519029tmp
CL> jenkins 37958 0.0 0.5 4946924 567908 ? Sl Jan23 2:01
/home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath
/home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
-Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL>
-Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL>
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository
-U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> asf999 59270 0.0 0.0 16576 2088 pts/0 S+ 05:39 0:00 grep
--color=auto CXF
>> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <c...@apache.org> wrote:
>> Here is some data from H24, which also contains many broken CXF jobs (not
>> Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF
>> builds, not karaf builds as previously noted. Copying dev@CXF since this
>> build seems to be related to ongoing node problems.
>> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I
>> have no idea what these are from, but they appear to be docker related.
>> asf999@asf924:~$ ps wwaxu | grep CXF
>> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar
>> org.apache.karaf.main.Main
>> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52
>> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
>> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar
>>
>> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire
>> 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp
>> surefire_1029401344835483973390tmp
>> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar
>> org.apache.karaf.main.Main
>> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52
>> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
>> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar
>>
>> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire
>> 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp
>> surefire_359166059833874743459tmp
>> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar
>> org.apache.karaf.main.Main
>> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar
>> org.apache.karaf.main.Main
>> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar
>> org.apache.karaf.main.Main
>> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar
>> org.apache.karaf.main.Main
>> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar
>> org.apache.karaf.main.Main
>> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar
>> org.apache.karaf.main.Main
>> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31
>> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
>> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed
>>
>> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
>>
>> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances
>>
>> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
>>
>> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
>>
>> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data
>>
>> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc
>>
>> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties
>> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
>> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar
>> org.apache.karaf.main.Main
>> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45
>> /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath
>> /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
>> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
>> -Dmaven.home=/home/jenkins/tools/maven/latest3
>> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
>> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14
>> org.codehaus.plexus.classworlds.launcher.Launcher
>> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository
>> -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
>> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep
>> --color=auto CXF
>>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <chr...@apache.org> wrote:
>>> Here is some data from H41, which was rebooted last night and ran out of
>>> threads today. https://paste.apache.org/lkmpq
>>> In this case it looks like Karaf was still stuck/broken even though there
>>> were no builds running on H41 at the time I investigated.
>>> -Chris T.
>>> #asfinfra
>>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>>> Is there some way we can improve the visibility into disk usage on the
>>>>> build nodes? How full they are? And what projects are taking up space?
>>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>>> somewhere?
>>>> There are two Jenkins plugins that help with this situation
>>>> - CloudBees Disk Usage Simple
>>>> - Percentage Disk Space Node Column
>>>> Could either/both of them be an option to install?
>>>> regards,
>>>> Mick
