Hello everyone, I have a question concerning stopping Flink streaming processes that run in a detached Yarn session.
Here's what we do: We start a Yarn session via yarn-session.sh -n 8 -d -jm 4096 -tm 10000 -s 10 -qu flink_queue Then, we start our Flink streaming application via flink run -p 65 -c SomeClass some.jar > /dev/null 2>&1 & The problem occurs when we stop the application. If we stop the Flink application with flink cancel <JOB_ID> and then kill the yarn application with yarn application -kill <APPLICATION_ID> everything is fine. But what we expected was that when we only kill the yarn application without specifically canceling the Flink job before, the Flink job will stay lingering on the machine and use resources until it is killed manually via its process id. One thing that we tried was to stop using ephemeral ports for the application-manager, namely we set yarn.application-master.port specifically to some port number, but the problem remains: Killing the yarn application does not kill the corresponding Flink job. Does anyone have an idea about this? Any help is greatly appreciated :-) By the way, our application reads data from a Kafka queue and writes it into HDFS, maybe this is also important to know. Thank you and best regards Konstantin -- Konstantin Gregor * konstantin.gre...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082
