Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10578300
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -17,28 +17,24 @@
package org.apache.spark.scheduler
+import scala.collection.mutable.HashMap
+
import java.io._
import java.util.zip.{GZIPInputStream, GZIPOutputStream}
-import scala.collection.mutable.HashMap
-
import org.apache.spark._
import org.apache.spark.executor.ShuffleWriteMetrics
-import org.apache.spark.rdd.RDD
-import org.apache.spark.rdd.RDDCheckpointData
+import org.apache.spark.rdd.{RDD, RDDCheckpointData}
import org.apache.spark.storage._
-import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType,
TimeStampedHashMap}
+import org.apache.spark.util.BoundedHashMap
private[spark] object ShuffleMapTask {
// A simple map between the stage id to the serialized byte array of a
task.
// Served as a cache for task serialization because serialization can be
// expensive on the master node if it needs to launch thousands of tasks.
- val serializedInfoCache = new TimeStampedHashMap[Int, Array[Byte]]
-
- // TODO: This object shouldn't have global variables
- val metadataCleaner = new MetadataCleaner(
- MetadataCleanerType.SHUFFLE_MAP_TASK,
serializedInfoCache.clearOldValues, new SparkConf)
+ val MAX_CACHE_SIZE = 100
+ val serializedInfoCache = new BoundedHashMap[Int,
Array[Byte]](MAX_CACHE_SIZE, true)
--- End diff --
@pwendell Wait, actually isn't that right? Correct me if I'm wrong, but
once a ShuffleDependency or an RDD goes out of scope, then all associated
stages can be safely removed, since to run tasks in the Stage you need this
information. This suggests if we keep track of a list of stage IDs in both
ShuffleDependency and RDD, then we can clean up this map (and the one in
ResultTask) explicitly.
@tdas, looks like this map only concerns the master, so I don't think we
need to notify the workers of stage completion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
