Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/1165#discussion_r14722107
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
* to the BlockManager as an iterator and expect to read it back
later. This is because
* we may end up dropping a partition from memory store before
getting it back, e.g.
* when the entirety of the RDD does not fit in memory. */
- val elements = new ArrayBuffer[Any]
- elements ++= values
- updatedBlocks ++= blockManager.put(key, elements, storageLevel,
tellMaster = true)
- elements.iterator.asInstanceOf[Iterator[T]]
+
--- End diff --
Looks like a code changed a bit since I last reviewed.
My point was rather simple : if we do not have offheap/disk enabled in the
storage level - what is the behavioral expectation in case size estimation says
it cant be cached due to memory limit ?
1) Do we abort the task (so that it might be cached on some other node),
2) Do we continue with it un-cached in memory (which seems to be the
approach after the recent changes) - which is a violation of the StorageLevel
contract.
3) Do we attempt to cache in memory anyway and risk OOM (and so (1) ).
Also note that the size estimation is just estimation and can be off mark -
so it is possible for the iterator to be cache-able, and yet estimated to be
not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
