GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/3321
[SPARK-4393] Fix memory leak in ConnectionManager ACK timeout TimerTasks;
use HashedWheelTimer (For branch-1.1)
This patch is intended to fix a subtle memory leak in ConnectionManager's
ACK timeout TimerTasks: in the old code, each TimerTask held a reference to the
message being sent and a cancelled TimerTask won't necessarily be
garbage-collected until it's scheduled to run, so this caused huge buildups of
messages that weren't garbage collected until their timeouts expired, leading
to OOMs.
This patch addresses this problem by capturing only the message ID in the
TimerTask instead of the whole message, and by keeping a WeakReference to the
promise in the TimerTask. I've also modified this code to use Netty's
HashedWheelTimer, whose performance characteristics should be better for this
use-case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark
connection-manager-timeout-bugfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3321.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3321
----
commit 786af9161b869b224f9ef314ba4b915a563acfb4
Author: Kousuke Saruta <[email protected]>
Date: 2014-11-17T19:38:35Z
Fixed memory leak issue of ConnectionManager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
