Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10835#discussion_r50642832
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -39,68 +37,144 @@ import org.apache.spark.util.Utils
*
* So, when adding new fields, take into consideration that the whole
object can be serialized for
* shipping off at any time to consumers of the SparkListener interface.
+ *
+ * @param initialAccums the initial set of accumulators that this
[[TaskMetrics]] depends on.
+ * Each accumulator in this initial set must be
uniquely named and marked
+ * as internal. Additional accumulators registered
later need not satisfy
+ * these requirements.
*/
@DeveloperApi
-class TaskMetrics extends Serializable {
+class TaskMetrics(initialAccums: Seq[Accumulator[_]]) extends Serializable
{
+
+ import InternalAccumulator._
+
+ // Needed for Java tests
+ def this() {
+ this(InternalAccumulator.create())
+ }
+
/**
- * Host's name the task runs on
+ * All accumulators registered with this task.
*/
- private var _hostname: String = _
- def hostname: String = _hostname
- private[spark] def setHostname(value: String) = _hostname = value
+ private val accums = new ArrayBuffer[Accumulable[_, _]]
+ accums ++= initialAccums
/**
- * Time taken on the executor to deserialize this task
+ * A map for quickly accessing the initial set of accumulators by name.
*/
- private var _executorDeserializeTime: Long = _
- def executorDeserializeTime: Long = _executorDeserializeTime
- private[spark] def setExecutorDeserializeTime(value: Long) =
_executorDeserializeTime = value
+ private val initialAccumsMap: Map[String, Accumulator[_]] = {
+ val map = new mutable.HashMap[String, Accumulator[_]]
+ initialAccums.foreach { a =>
+ assert(a.name.isDefined, "initial accumulators passed to TaskMetrics
must be named")
+ val name = a.name.get
--- End diff --
You could do `val name = a.name.getOrElse { throw new
IllegalArgumentException("...") }`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
