Hi, Just found out that ShuffleMapTask has transient locs and preferredLocs attributes which means that when ShuffleMapTask is serialized (as a broadcast variable) the information is gone.
Does this mean that the attributes could have not been defined at all since Spark uses SortShuffleManager (and BlockManagerMaster on the driver) to track the shuffle locations (MapStatuses)? Is my understanding correct? What am I missing? (I'm exploring shuffle system currently and would appreciate comments a lot!) Thanks! Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
