ahshahid commented on PR #50757: URL: https://github.com/apache/spark/pull/50757#issuecomment-2845911576
A stage is inDerterministic or not should not depend on it's parent stage but only on the behaviour of the partitioner of the current stage. On Thu, May 1, 2025, 2:23 PM Asif Shahid ***@***.***> wrote: > I am out right now.. will check back.. but what I m trying to say is that > a map stage should be marked inDeterminate iff the partitioner is using > indeterministic Val. And afaik only SQL has metadata to convey that info... > I don't know in what situation a non SQL rdd can get that info and that too > correctly. > > On Thu, May 1, 2025, 2:12 PM Attila Zsolt Piros ***@***.***> > wrote: > >> ***@***.**** commented on this pull request. >> ------------------------------ >> >> On core/src/main/scala/org/apache/spark/rdd/RDD.scala >> <https://github.com/apache/spark/pull/50757#discussion_r2070814828>: >> >> Please check this lines in RDD.scala: >> >> https://github.com/apache/spark/blob/085bfc21ce0e162eb15257649ec1a3dc6f25c5be/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L2111-L2114 >> >> This is where INDETERMINATE level is inherited from one stage to another. >> >> — >> Reply to this email directly, view it on GitHub >> <https://github.com/apache/spark/pull/50757#discussion_r2070814828>, or >> unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AC6XG2HINSI2JX6BTVEQSMD24KE4TAVCNFSM6AAAAAB4DOZR32VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDQMJQG42DANRYGM> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
