Hello, I checked PR [1] and found many build failures on H41 during 'git' command execution [2]. The last failure was OOM [3]. Finally build was successful using H29 [4].
1. https://github.com/apache/cxf/pull/631 2. https://builds.apache.org/job/CXF-Trunk-PR/1389/console 3. https://builds.apache.org/job/CXF-Trunk-PR/1390/console 4. https://builds.apache.org/job/CXF-Trunk-PR/1391/ On 1/27/20, Chris Lambertus <c...@apache.org> wrote: > Here is some data from H24, which also contains many broken CXF jobs (not > Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF > builds, not karaf builds as previously noted. Copying dev@CXF since this > build seems to be related to ongoing node problems. > > > Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I > have no idea what these are from, but they appear to be docker related. > > > asf999@asf924:~$ ps wwaxu | grep CXF > jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar > org.apache.karaf.main.Main > jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 > /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar > /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar > /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire > 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp > surefire_1029401344835483973390tmp > jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar > org.apache.karaf.main.Main > jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 > /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar > /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar > /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire > 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp > surefire_359166059833874743459tmp > jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar > org.apache.karaf.main.Main > jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar > org.apache.karaf.main.Main > jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar > org.apache.karaf.main.Main > jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar > org.apache.karaf.main.Main > jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar > org.apache.karaf.main.Main > jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar > org.apache.karaf.main.Main > jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 > /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java > -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed > -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext > -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances > -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 > -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 > -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data > -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc > -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties > -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp > /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar > org.apache.karaf.main.Main > jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 > /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath > /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar > -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf > -Dmaven.home=/home/jenkins/tools/maven/latest3 > -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native > -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 > org.codehaus.plexus.classworlds.launcher.Launcher > -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository > -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks > asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep > --color=auto CXF > > >> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <chr...@apache.org> >> wrote: >> >> Here is some data from H41, which was rebooted last night and ran out of >> threads today. https://paste.apache.org/lkmpq >> >> In this case it looks like Karaf was still stuck/broken even though there >> were no builds running on H41 at the time I investigated. >> >> -Chris T. >> #asfinfra >> >> >> On 1/24/20 4:26 AM, Mick Semb Wever wrote: >>> >>>> Is there some way we can improve the visibility into disk usage on the >>>> build nodes? How full they are? And what projects are taking up space? >>>> Does jenkins provide this info? Or could infra dump a `du …` report >>>> somewhere? >>> >>> >>> There are two Jenkins plugins that help with this situation >>> - CloudBees Disk Usage Simple >>> - Percentage Disk Space Node Column >>> >>> Could either/both of them be an option to install? >>> >>> regards, >>> Mick >> > >
