Hi, Just noticed that TaskContext#getPartitionId [1] is not used and moreover the scaladoc is incorrect:
"It will return 0 if there is no active TaskContext for cases like local execution." since there are no local execution. (I've seen the comment in the code before but can't find it now). The reason to remove it is that Structured Streaming is giving new birth to the method in ForeachSink [2] which may look like a "resurrection". There's simply TaskContext.get.partitionId. What do you think? [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/TaskContext.scala#L41 [2] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ForeachSink.scala#L50 Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
