Are you running in HA mode? If yes, that's the expected behaviour at the moment, because the ZooKeeper data is only cleaned up on a terminal state (FINISHED, FAILED, CANCELLED). You have to specify separate ZooKeeper root paths via "recovery.zookeeper.path.root". There is an issue which should be fixed for 1.2 to make this configurable in an easy way.
On Tue, Jul 12, 2016 at 1:28 PM, Konstantin Gregor <konstantin.gre...@tngtech.com> wrote: > Hello everyone, > > I have a question concerning stopping Flink streaming processes that run > in a detached Yarn session. > > Here's what we do: We start a Yarn session via > yarn-session.sh -n 8 -d -jm 4096 -tm 10000 -s 10 -qu flink_queue > > Then, we start our Flink streaming application via > flink run -p 65 -c SomeClass some.jar > /dev/null 2>&1 & > > The problem occurs when we stop the application. > If we stop the Flink application with > flink cancel <JOB_ID> > and then kill the yarn application with > yarn application -kill <APPLICATION_ID> > everything is fine. > But what we expected was that when we only kill the yarn application > without specifically canceling the Flink job before, the Flink job will > stay lingering on the machine and use resources until it is killed > manually via its process id. > > One thing that we tried was to stop using ephemeral ports for the > application-manager, namely we set yarn.application-master.port > specifically to some port number, but the problem remains: Killing the > yarn application does not kill the corresponding Flink job. > > Does anyone have an idea about this? Any help is greatly appreciated :-) > By the way, our application reads data from a Kafka queue and writes it > into HDFS, maybe this is also important to know. > > Thank you and best regards > > Konstantin > -- > Konstantin Gregor * konstantin.gre...@tngtech.com > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082
