GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/980
SPARK-2034. KafkaInputDStream doesn't close resources and may prevent JVM
shutdown
Tobias noted today on the mailing list:
========
I am trying to use Spark Streaming with Kafka, which works like a
charm â except for shutdown. When I run my program with "sbt
run-main", sbt will never exit, because there are two non-daemon
threads left that don't die.
I created a minimal example at
<https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-kafkadoesntshutdown-scala>.
It starts a StreamingContext and does nothing more than connecting to
a Kafka server and printing what it receives. Using the `future
Unknown macro: { ... }
` construct, I shut down the StreamingContext after some seconds and
then print the difference between the threads at start time and at end
time. The output can be found at
<https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output1>.
There are a number of threads remaining that will prevent sbt from
exiting.
When I replace `KafkaUtils.createStream(...)` with a call that does
exactly the same, except that it calls `consumerConnector.shutdown()`
in `KafkaReceiver.onStop()` (which it should, IMO), the output is as
shown at
<https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output2>.
Does anyone have any idea what is going on here and why the program
doesn't shut down properly? The behavior is the same with both kafka
0.8.0 and 0.8.1.1, by the way.
========
Something similar was noted last year:
http://mail-archives.apache.org/mod_mbox/spark-dev/201309.mbox/%[email protected]%3E
KafkaInputDStream doesn't close `ConsumerConnector` in `onStop()`, and does
not close the `Executor` it creates. The latter leaves non-daemon threads and
can prevent the JVM from shutting down even if streaming is closed properly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-2034
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/980.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #980
----
commit 2d579a83b792d91a67ba22eefe5c5032386149a2
Author: Sean Owen <[email protected]>
Date: 2014-06-05T10:08:36Z
Close ConsumerConnector in onStop; shutdown() the local Executor that is
created so that its threads stop when done; close the Zookeeper client even on
exception; fix a few typos; log exceptions that otherwise vanish
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
