Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related. asf999@asf924:~$ ps wwaxu | grep CXF jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF > On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <chr...@apache.org> wrote: > > Here is some data from H41, which was rebooted last night and ran out of > threads today. https://paste.apache.org/lkmpq > > In this case it looks like Karaf was still stuck/broken even though there > were no builds running on H41 at the time I investigated. > > -Chris T. > #asfinfra > > > On 1/24/20 4:26 AM, Mick Semb Wever wrote: >> >>> Is there some way we can improve the visibility into disk usage on the >>> build nodes? How full they are? And what projects are taking up space? >>> Does jenkins provide this info? Or could infra dump a `du …` report >>> somewhere? >> >> >> There are two Jenkins plugins that help with this situation >> - CloudBees Disk Usage Simple >> - Percentage Disk Space Node Column >> >> Could either/both of them be an option to install? >> >> regards, >> Mick >
