Re: [ANNOUNCE] Announcing Apache Spark Kubernetes Operator 0.1.0

Thank you, Mridul. The typo is fixed like the following. - Support Apache Spark 3.5 and 4.0 + Support Apache Spark 3.5 and 4.0 (RC4) Dongjoon. On 2025/05/08 15:01:48 Mridul Muralidharan wrote: > I had not checked the release. > > The release notes mention that Apache Spark 4.0 is supported - wh

[DISCUSS][SPARK SQL] SPARK-51710: Using Dataframe.dropDuplicates with an empty array as argument behaves "unexpectedly"

Hello everyone, Following the creation of this PR and the discussion in the thread. What do you think about the behavior described here: When using PySpark DataFrame.dropDuplicates with an empty array as the > subset argument, the resulting DataFrame

[VOTE][RESULT] SPIP: Add geospatial types to Spark

The vote passes with 19 +1s (12 binding +1s) and one +0. Thank you to all that have participated in the discussion and the vote and for all the feedback! (* = binding) +1: - DB Tsai * - Szehon Ho - Milan Stefanovic - Bjørn Jørgensen - Reynold Xin * - Wenchen Fan * - Denny Lee - Yuming Wang * - Xi

Re: Extending Spark with a custom ExternalClusterManager

Hi all, I just created a JIRA ticket and a work in progress PR. Here is the link to the JIRA ticket - https://issues.apache.org/jira/browse/SPARK-52041 Here is the link to the GitHub PR - https://github.com/apache/spark/pull/50770 I kindly ask for feedback. Kind regards On Wed, Feb 19, 2025 at

Re: [ANNOUNCE] Announcing Apache Spark Kubernetes Operator 0.1.0

I had not checked the release. The release notes mention that Apache Spark 4.0 is supported - which has not yet been released. While I don’t expect drastic changes - and most likely the support which will continue to work, the messaging is not accurate - Mridul On Wed, May 7, 2025 at 8:54 PM Do