Can you clarify what you mean by "used for all stages"?
OutputCommitCoordinator RPCs should only be initiated through
SparkHadoopMapRedUtil.commitTask(), so while the OutputCommitCoordinator
doesn't make a distinction between ShuffleMapStages and ResultStages
there still should not be a performance penalty for this because the
extra rounds of RPCs should only be performed when necessary.
On 8/11/15 2:25 AM, Jeff Zhang wrote:
As my understanding, OutputCommitCoordinator should only be necessary
for ResultStage (especially for ResultStage with hdfs write), but
currently it is used for all the stages. Is there any reason for that ?
--
Best Regards
Jeff Zhang
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
