Andrew Milkowski created SPARK-19364:
----------------------------------------
Summary: Some Blocks in Storage Persists Forever
Key: SPARK-19364
URL: https://issues.apache.org/jira/browse/SPARK-19364
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.0.2
Environment: ubuntu unix
spar 2.0.2
application is java
Reporter: Andrew Milkowski
running standard kinesis stream ingestion with a java spark app and creating
dstream after running for some time some block streams seem to persist forever
and never cleaned up and this eventually leads to memory depletion on workers
we even tried cleaning RDD's with the following:
cleaner = ssc.sparkContext().sc().cleaner().get();
filtered.foreachRDD(new VoidFunction<JavaRDD<String>>() {
@Override
public void call(JavaRDD<String> rdd) throws Exception {
cleaner.doCleanupRDD(rdd.id(), true);
}
});
despite above blocks do persis still, this can be seen in spark admin UI
for instance
input-0-1485362233945 1 ip-<>:34245 Memory Serialized 1442.5
KB
above block stays and is never cleaned up
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
