carlmartin created SPARK-17233:
----------------------------------
Summary: Shuffle file will be left over the capacity when dynamic
schedule is enabled in a long running case.
Key: SPARK-17233
URL: https://issues.apache.org/jira/browse/SPARK-17233
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.0.0, 1.6.2, 1.5.2
Reporter: carlmartin
When I execute some sql statement periodically in the long running
thriftserver, I found the disk device will be full after about one week later.
After check the file on linux, I found so many shuffle files left on the
block-mgr dir whose shuffle stage had finished long time ago.
Finally I find when it's need to clean shuffle file, driver will total each
executor to do the ShuffleClean. But when dynamic schedule is enabled, executor
will be down itself and executor can't clean its shuffle file, then file was
left.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
