[
https://issues.apache.org/jira/browse/SPARK-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644875#comment-15644875
]
Shivaram Venkataraman commented on SPARK-17822:
-----------------------------------------------
Yeah just to provide some more context, every time an object is added to the
JVMObjectTracker we correspondingly make a note of it on the R side (in a map
called .validJobjs[1]).
This map and correspondingly the JVM side objects should get cleared up when
the object gets GC'd R[2].
I think there are two possibilities here - one is the R side actually is
holding on to these references and not clearing them. The other is that the GC
messages from R to JVM is somehow not working as expected.
[~yhuai] If we give you some debug scripts can you run them on the R side
before the crash ?
[1]
https://github.com/apache/spark/blob/daa975f4bfa4f904697bf3365a4be9987032e490/R/pkg/R/jobj.R#L43
[2]
https://github.com/apache/spark/blob/daa975f4bfa4f904697bf3365a4be9987032e490/R/pkg/R/jobj.R#L90
> JVMObjectTracker.objMap may leak JVM objects
> --------------------------------------------
>
> Key: SPARK-17822
> URL: https://issues.apache.org/jira/browse/SPARK-17822
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Reporter: Yin Huai
> Attachments: screenshot-1.png
>
>
> JVMObjectTracker.objMap is used to track JVM objects for SparkR. However, we
> observed that JVM objects that are not used anymore are still trapped in this
> map, which prevents those object get GCed.
> Seems it makes sense to use weak reference (like persistentRdds in
> SparkContext).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
