Hi Sean, Can you elaborate on " it's actually used by Spark"? Where exactly? I'd like to be corrected.
What about the scaladoc? Since the method's a public API, I think it should be fixed, shouldn't it? Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sat, Jan 14, 2017 at 6:03 PM, Sean Owen <so...@cloudera.com> wrote: > It doesn't strike me as something that's problematic to use. There are a > thousand things in the API that, maybe in hindsight, could have been done > differently, but unless something is bad practice or superseded by another > superior mechanism, it's probably not worth the bother for maintainers or > users. I don't see much problem with this, and it's actually used by Spark. > > On Sat, Jan 14, 2017 at 3:55 PM Jacek Laskowski <ja...@japila.pl> wrote: >> >> Hi, >> >> Yes, correct. I was too forceful in discouraging people using it. I think >> @deprecated would be a right direction. >> >> What should be the next step? I think I should file an JIRA so it's in a >> release notes. Correct? >> >> I was very surprised to have noticed its resurrection in the very latest >> module of Spark - Structured Streaming - that will be an inspiration for >> others to learn Spark. >> >> Jacek >> >> On 14 Jan 2017 12:48 p.m., "Mridul Muralidharan" <mri...@gmail.com> wrote: >>> >>> Since TaskContext.getPartitionId is part of the public api, it cant be >>> removed as user code can be depending on it (unless we go through a >>> deprecation process for it). >>> >>> Regards, >>> Mridul >>> >>> >>> On Sat, Jan 14, 2017 at 2:02 AM, Jacek Laskowski <ja...@japila.pl> wrote: >>> > Hi, >>> > >>> > Just noticed that TaskContext#getPartitionId [1] is not used and >>> > moreover the scaladoc is incorrect: >>> > >>> > "It will return 0 if there is no active TaskContext for cases like >>> > local execution." >>> > >>> > since there are no local execution. (I've seen the comment in the code >>> > before but can't find it now). >>> > >>> > The reason to remove it is that Structured Streaming is giving new >>> > birth to the method in ForeachSink [2] which may look like a >>> > "resurrection". >>> > >>> > There's simply TaskContext.get.partitionId. >>> > >>> > What do you think? >>> > >>> > [1] >>> > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/TaskContext.scala#L41 >>> > [2] >>> > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ForeachSink.scala#L50 >>> > >>> > Pozdrawiam, >>> > Jacek Laskowski >>> > ---- >>> > https://medium.com/@jaceklaskowski/ >>> > Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark >>> > Follow me at https://twitter.com/jaceklaskowski >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
