Re:[VOTE] Release Spark 4.0.0 (RC2)

-1 https://github.com/apache/spark/pull/50112 should be merged before release. At 2025-03-01 15:25:06, "Wenchen Fan" wrote: Please vote on releasing the following candidate as Apache Spark version 4.0.0. The vote is open until March 5 (PST) and passes if a majority +1 PMC votes are cas

Re: [VOTE] Release Spark 4.0.0 (RC2)

Thanks for bringing up these blockers! I know RC2 isn’t fully ready yet, but with over 70 commits since RC1, it’s time to have a new RC so people can start testing the latest changes. Please continue testing and keep the feedback coming! On Mon, Mar 3, 2025 at 6:06 PM beliefer wrote: > -1 > > ht

Re: Seek for consensus on landing Spark Connect implementation for transformWithState in Spark 4.0.0

+1 - Would be great to get this into the Spark 4.0 release. Thanks, Anish On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim wrote: > Hi dev, > > We are going to introduce a new API named `transformWithState` for > streaming query, which allows users to perform more complex stateful > operation in use

[DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

Hi dev, This is a spin-up of the original thread "Deprecating and banning `spark.databricks.*` config from Apache Spark repository". (link ) >From the original thread, we decided to deprecate the config in Spark 3.5.5 and remove th

Seek for consensus on landing Spark Connect implementation for transformWithState in Spark 4.0.0

Hi dev, We are going to introduce a new API named `transformWithState` for streaming query, which allows users to perform more complex stateful operation in user function, with lot simpler code compared to `flatMapGroupsWithState` (and `applyInPandasWithState`). The target version has been Spark