+1 - Would be great to get this into the Spark 4.0 release. Thanks, Anish
On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Hi dev, > > We are going to introduce a new API named `transformWithState` for > streaming query, which allows users to perform more complex stateful > operation in user function, with lot simpler code compared to > `flatMapGroupsWithState` (and `applyInPandasWithState`). > > The target version has been Spark 4.0.0 and we track this project as a > major one for Spark 4. We push most planned features into Spark 4.0.0, > except Spark Connect support. > > The PRs for Spark Connect support are merged into Spark 4.1 branch, but > I'm seeking the voice whether we can introduce Spark Connect support to > Spark 4.0.0. > > I understand this arrives a bit late, but since the API is something > backed by a huge effort and I foresee this new API to replace the usage of > flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to make > sure we don't push users back to wait for another 6+ months to use this in > Spark Connect. > > Would love to hear your thoughts. > > Thanks, > Jungtaek Lim (HeartSaVioR) >
