Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10552777
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -17,28 +17,24 @@
package org.apache.spark.scheduler
+import scala.collection.mutable.HashMap
+
import java.io._
import java.util.zip.{GZIPInputStream, GZIPOutputStream}
-import scala.collection.mutable.HashMap
-
import org.apache.spark._
import org.apache.spark.executor.ShuffleWriteMetrics
-import org.apache.spark.rdd.RDD
-import org.apache.spark.rdd.RDDCheckpointData
+import org.apache.spark.rdd.{RDD, RDDCheckpointData}
import org.apache.spark.storage._
-import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType,
TimeStampedHashMap}
+import org.apache.spark.util.BoundedHashMap
private[spark] object ShuffleMapTask {
// A simple map between the stage id to the serialized byte array of a
task.
// Served as a cache for task serialization because serialization can be
// expensive on the master node if it needs to launch thousands of tasks.
- val serializedInfoCache = new TimeStampedHashMap[Int, Array[Byte]]
-
- // TODO: This object shouldn't have global variables
- val metadataCleaner = new MetadataCleaner(
- MetadataCleanerType.SHUFFLE_MAP_TASK,
serializedInfoCache.clearOldValues, new SparkConf)
+ val MAX_CACHE_SIZE = 100
+ val serializedInfoCache = new BoundedHashMap[Int,
Array[Byte]](MAX_CACHE_SIZE, true)
--- End diff --
Actually, this unrelated to shuffle, because caches for both,
ShuffleMapTasks and ResultsTasks, need to be cleaned. One nifty way I can think
of keeping track of the lowest stage ID S in the DAGScheduler such that all
stages with id <= S have completed. If that is somehow communicated to all the
workers, then cleanup can be done. Though, there is a obvious problem of what
happens when the id rolls around Integer.MAX_VALUE (though I am not sure how
serious a problem that is).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
