Hi Jungtaek, We are already in RC2 for 4.0, right ? A bit too late for this IMO - we can always introduce it in 4.1
Regards, Mridul On Tue, Mar 4, 2025 at 7:22 AM Herman van Hovell <her...@databricks.com.invalid> wrote: > +1 > > On Tue, Mar 4, 2025 at 2:07 AM Anish Shrigondekar > <anish.shrigonde...@databricks.com.invalid> wrote: > >> +1 - Would be great to get this into the Spark 4.0 release. >> >> Thanks, >> Anish >> >> On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> >> wrote: >> >>> Hi dev, >>> >>> We are going to introduce a new API named `transformWithState` for >>> streaming query, which allows users to perform more complex stateful >>> operation in user function, with lot simpler code compared to >>> `flatMapGroupsWithState` (and `applyInPandasWithState`). >>> >>> The target version has been Spark 4.0.0 and we track this project as a >>> major one for Spark 4. We push most planned features into Spark 4.0.0, >>> except Spark Connect support. >>> >>> The PRs for Spark Connect support are merged into Spark 4.1 branch, but >>> I'm seeking the voice whether we can introduce Spark Connect support to >>> Spark 4.0.0. >>> >>> I understand this arrives a bit late, but since the API is something >>> backed by a huge effort and I foresee this new API to replace the usage of >>> flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to make >>> sure we don't push users back to wait for another 6+ months to use this in >>> Spark Connect. >>> >>> Would love to hear your thoughts. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>
