Hi, I have two machines running yarn and samza. They are samza 0.9.1 and hadoop 2.6.0.
I run the kill-all.sh I recently wrote and calls the kill-yarn-job.sh. This is the output: java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dlog4j.configuration=file:bin/log4j-console.xml -Dsamza.log.dir=/opt/jobs -Djava.io.tmpdir=/opt/jobs/tmp -Xmx768M -XX:+PrintGCDateStamps -Xloggc:/opt/jobs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp /opt/hadoop-2.6.0/conf:/opt/jobs/lib/redirect-0.0.1.jar:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar org.apache.hadoop.yarn.client.cli.ApplicationCLI application -kill application_1442908447829_0001 2015-09-22 10:02:46 RMProxy [INFO] Connecting to ResourceManager at kfk-samza01/192.168.15.92:8032 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2015-09-22 10:02:46 NativeCodeLoader [WARN] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Killing application application_1442908447829_0001 2015-09-22 10:02:47 YarnClientImpl [INFO] Killed application application_1442908447829_0001 java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb7u1) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dlog4j.configuration=file:bin/log4j-console.xml -Dsamza.log.dir=/opt/jobs -Djava.io.tmpdir=/opt/jobs/tmp -Xmx768M -XX:+PrintGCDateStamps -Xloggc:/opt/jobs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp /opt/hadoop-2.6.0/conf:/opt/jobs/lib/redirect-0.0.1.jar:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar org.apache.hadoop.yarn.client.cli.ApplicationCLI application -kill application_1442908447829_0002 2015-09-22 10:02:49 RMProxy [INFO] Connecting to ResourceManager at kfk-samza01/192.168.15.92:8032 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/jobs/lib/redirect-0.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/jobs/lib/samzafroga-0.0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2015-09-22 10:02:49 NativeCodeLoader [WARN] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Killing application application_1442908447829_0002 2015-09-22 10:02:50 YarnClientImpl [INFO] Killed application application_1442908447829_0002 When I run ps -fe |grep java I see this: root 9542 1 3 09:54 pts/0 00:00:21 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/opt/hadoop-2.6.0/logs -Dyarn.log.dir=/opt/hadoop-2.6.0/logs -Dhadoop.log.file=yarn-root-resourcemanager-kfk-samza01.log -Dyarn.log.file=yarn-root-resourcemanager-kfk-samza01.log -Dyarn.home.dir=/opt/hadoop-2.6.0 -Dhadoop.home.dir=/opt/hadoop-2.6.0 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop-2.6.0/lib/native -classpath /opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/share/hadoop/common/lib/*:/opt/hadoop-2.6.0/share/hadoop/common/*:/opt/hadoop-2.6.0/share/hadoop/hdfs:/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/opt/hadoop-2.6.0/share/hadoop/hdfs/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/conf/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager root 9814 1 4 09:54 ? 00:00:24 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_nodemanager -Xmx1000m -server -Dhadoop.log.dir=/opt/hadoop-2.6.0/logs -Dyarn.log.dir=/opt/hadoop-2.6.0/logs -Dhadoop.log.file=yarn-root-nodemanager-kfk-samza01.log -Dyarn.log.file=yarn-root-nodemanager-kfk-samza01.log -Dyarn.home.dir=/opt/hadoop-2.6.0 -Dhadoop.home.dir=/opt/hadoop-2.6.0 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/hadoop-2.6.0/lib/native -classpath /opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/conf:/opt/hadoop-2.6.0/share/hadoop/common/lib/*:/opt/hadoop-2.6.0/share/hadoop/common/*:/opt/hadoop-2.6.0/share/hadoop/hdfs:/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/opt/hadoop-2.6.0/share/hadoop/hdfs/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-2.6.0/share/hadoop/mapreduce/*:/opt/hadoop-2.6.0/share/hadoop/yarn/*:/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*:/opt/hadoop-2.6.0/conf/nm-config/log4j.properties org.apache.hadoop.yarn.server.nodemanager.NodeManager root 10271 10268 0 09:54 ? 00:00:05 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Dsamza.container.name=samza-application-master -Dsamza.log.dir=/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0002/container_1442908447829_0002_01_000001 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/tmp -Xmx768M -XX:+PrintGCDateStamps -Xloggc:/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0002/container_1442908447829_0002_01_000001/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp /opt/hadoop-2.6.0/conf:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-core-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-databind-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-dataformat-smile-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-jaxrs-json-provider-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/jackson-module-jaxb-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/nxtBroker-0.0.1.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0002/container_1442908447829_0002_01_000001/__package/lib/nxtBroker-0.0.1-jar-with-dependencies.jar org.apache.samza.job.yarn.SamzaAppMaster root 10346 10344 0 09:54 ? 00:00:04 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Dsamza.container.name=samza-application-master -Dsamza.log.dir=/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0001/container_1442908447829_0001_01_000001 -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/tmp -Xmx768M -XX:+PrintGCDateStamps -Xloggc:/opt/hadoop-2.6.0/logs/userlogs/application_1442908447829_0001/container_1442908447829_0001_01_000001/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp /opt/hadoop-2.6.0/conf:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-core-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-databind-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-dataformat-smile-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-jaxrs-json-provider-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/jackson-module-jaxb-annotations-2.6.0.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/nxtBroker-0.0.1.jar:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1442908447829_0001/container_1442908447829_0001_01_000001/__package/lib/nxtBroker-0.0.1-jar-with-dependencies.jar org.apache.samza.job.yarn.SamzaAppMaster As you can see the proceses are still there. In the web application the appear as KILLED. Thanks, Jordi -----Mensaje original----- De: Yan Fang [mailto:yanfang...@gmail.com] Enviado el: martes, 22 de septiembre de 2015 9:59 Para: dev@samza.apache.org Asunto: Re: process killing Hi Jordi, 1. Are you running the job in one machine yarn? or in a cluster? 2. what kind of the java process do you see after killing the yarn application? Because usually, when we run kill-yarn-job applicationId, we do kill all the processes (this is actually done by the Yarn). 3. Which version of Samza and Yarn are you using ? This matters sometimes. Thanks, Fang, Yan yanfang...@gmail.com On Tue, Sep 22, 2015 at 3:42 PM, Jordi Blasi Uribarri <jbl...@nextel.es> wrote: > Hi, > > I am currently developing solution using samza and in the development > process I need to constantly change the code and test in the system. > What I am seeing is that most of the times I kill a job using the > kill-yarn-job script the job gets killed according to the web > interface but I see the java process running. I also have seen that > the job was actually been executed, as I got messages in the far end > of the application. I have been manually killing these processes (kill -9 ) > but I have some questions: > > > - Is there a reason for the processes not to be killed. It was > not a matter of time as I could find them hours later. > > - I don’t know if there should be any other action performed to > completely clean the information or killing the process the hard way > is enough. > > - I am finding some memory consumption problems that I don’t know > if they are related with this. Maybe I will describe them in another > message. > > Thnaks, > > Jordi > ________________________________ > Jordi Blasi Uribarri > Área I+D+i > > jbl...@nextel.es > Oficina Bilbao > > [http://www.nextel.es/wp-content/uploads/Firma_Nextel_2015.png] >
