Hi, Here are PRs we are seeking for consensus to get in for 4.0.
PySpark: https://github.com/apache/spark/pull/49560 Scala: https://github.com/apache/spark/pull/49488 Thanks, Jungtaek Lim (HeartSaVioR) On Tue, Mar 4, 2025 at 11:06 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks. > > Can you point to a link or any further documentation please? > > Dr Mich Talebzadeh, > Architect | Data Science | Financial Crime | Forensic Analysis | GDPR > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > > > On Tue, 4 Mar 2025 at 13:22, Herman van Hovell > <her...@databricks.com.invalid> wrote: > >> +1 >> >> On Tue, Mar 4, 2025 at 2:07 AM Anish Shrigondekar >> <anish.shrigonde...@databricks.com.invalid> wrote: >> >>> +1 - Would be great to get this into the Spark 4.0 release. >>> >>> Thanks, >>> Anish >>> >>> On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>>> Hi dev, >>>> >>>> We are going to introduce a new API named `transformWithState` for >>>> streaming query, which allows users to perform more complex stateful >>>> operation in user function, with lot simpler code compared to >>>> `flatMapGroupsWithState` (and `applyInPandasWithState`). >>>> >>>> The target version has been Spark 4.0.0 and we track this project as a >>>> major one for Spark 4. We push most planned features into Spark 4.0.0, >>>> except Spark Connect support. >>>> >>>> The PRs for Spark Connect support are merged into Spark 4.1 branch, but >>>> I'm seeking the voice whether we can introduce Spark Connect support to >>>> Spark 4.0.0. >>>> >>>> I understand this arrives a bit late, but since the API is something >>>> backed by a huge effort and I foresee this new API to replace the usage of >>>> flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to make >>>> sure we don't push users back to wait for another 6+ months to use this in >>>> Spark Connect. >>>> >>>> Would love to hear your thoughts. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>>
