I tried adding shutdown hook to my code but it didn't help. Still same issue
On Fri, Nov 20, 2015 at 7:08 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Which Spark release are you using ? > > Can you pastebin the stack trace of the process running on your machine ? > > Thanks > > On Nov 20, 2015, at 6:46 PM, Vikram Kone <vikramk...@gmail.com> wrote: > > Hi, > I'm seeing a strange problem. I have a spark cluster in standalone mode. I > submit spark jobs from a remote node as follows from the terminal > > spark-submit --master spark://10.1.40.18:7077 --class com.test.Ping > spark-jobs.jar > > when the app is running , when I press ctrl-C on the console terminal, > then the process is killed and so is the app in the spark master UI. When I > go to spark master ui, i see that this app is in state Killed under > Completed applications, which is what I expected to see. > > Now, I created a shell script as follows to do the same > > #!/bin/bash > spark-submit --master spark://10.1.40.18:7077 --class com.test.Ping > spark-jobs.jar > echo $! > my.pid > > When I execute the shell script from terminal, as follows > > $> bash myscript.sh > > The application is submitted correctly to spark master and I can see it as > one of the running apps in teh spark master ui. But when I kill the process > in my terminal as follows > > $> ps kill $(cat my.pid) > > I see that the process is killed on my machine but the spark appliation is > still running in spark master! It doesn't get killed. > > I noticed one more thing that, when I launch the spark job via shell > script and kill the application from spark master UI by clicking on "kill" > next to the running application, it gets killed in spark ui but I still see > the process running in my machine. > > In both cases, I would expect the remote spark app to be killed and my > local process to be killed. > > Why is this happening? and how can I kill a spark app from the terminal > launced via shell script w.o going to the spark master UI? > > I want to launch the spark app via script and log the pid so i can monitor > it remotely > > thanks for the help > >
