Re: [VOTE] Release Spark 4.0.0 (RC1)

Is there also supposed to be a pyspark-client package released? Don't see that in the dist. On Fri, Feb 21, 2025 at 2:23 AM Hyukjin Kwon wrote: > Yeah I would like people to test this out. Let's wait bit more .. > On Fri, Feb 21, 2025 at 9:02 AM Mich Talebzadeh > wrote: > >> >>- >> >>Th

Re: 4.0.0 RC1 is coming

Hi Sorry for late reply, we identified another serious issue with the newly added Call Procedure, can we add it to the list? SPARK-51273: Spark Connect Call Procedure runs the procedure twice . I have a PR

Re: 4.0.0 RC1 is coming

Hi All, While testing new syntax CREATE FUNCTION ... RETURNS TABLE which was introduced recently, I have found that Spark fails with an internal error in 4.0.0-rc1, see https://issues.apache.org/jira/browse/SPARK-51289. I believe even if we don't fully support new feature, Spark shouldn't crash wi

Re: Behaviour of operators like Outer Join when using indeterministic joining keys seems to be full of contradictions

Checksum check as a fallback could indeed make sense. Good to have a catch all way here. > BTW, I'm working with my colleagues on this runtime checksum idea. We have something similar in our internal infra and I think we can create a working solution shortly @Wenchen Fan is there any SPIP/JIRA t

Re: [VOTE] Release Spark 4.0.0 (RC1)

-1, need a proper error message for not fully implemented feature: https://issues.apache.org/jira/browse/SPARK-51289 On Fri, Feb 21, 2025 at 4:11 PM Adam Binford wrote: > Is there also supposed to be a pyspark-client package released? Don't see > that in the dist. > > On Fri, Feb 21, 2025 at 2:2

Re: Behaviour of operators like Outer Join when using indeterministic joining keys seems to be full of contradictions

Hi, I have splitted my PR for inDeterministic Stage bug into 2 : The first one is for jira : https://issues.apache.org/jira/browse/SPARK-51016 The PR is: https://github.com/apache/spark/pull/50029 The above deals exclusively for the issue of incorrect result of Stage.isInDeterminate function. T

Re: [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration

Thank you all. This vote passed. I'll conclude the vote by sending a vote result email. Dongjoon. On 2025/02/21 00:37:07 John Zhuge wrote: > +1 (non-binding) > > John Zhuge > > > On Thu, Feb 20, 2025 at 1:52 PM Hussein Awala wrote: > > > +1 (non-binding) > > > > On Thu, Feb 20, 2025 at 10:4

[VOTE][RESULT] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration

The vote passes with 17 +1s (6 binding +1s). Thanks to all who helped with the vote! (* = binding) +1: - Dongjoon Hyun * - Mark Hamstra * - Yang Jie * - Sakthi - Wenchen Fan * - Angel - Mich Talebzadeh - Jungtaek Lim - Liang-Chi Hsieh * - Max Gekk - Peter Toth - Zhou Jiang - Reynold Xin * - Denny