Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1165#discussion_r14325571
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
* to the BlockManager as an iterator and expect to read it back
later. This is because
* we may end up dropping a partition from memory store before
getting it back, e.g.
* when the entirety of the RDD does not fit in memory. */
- val elements = new ArrayBuffer[Any]
- elements ++= values
- updatedBlocks ++= blockManager.put(key, elements, storageLevel,
tellMaster = true)
- elements.iterator.asInstanceOf[Iterator[T]]
+
--- End diff --
Maybe I'm not understanding what you're referring to? The whole point of
doing this is to make sure the partition doesn't cause an OOM if it's large.
This is a safety issue also for `MEMORY_ONLY`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
