+1 On Tue, Mar 4, 2025 at 2:07 AM Anish Shrigondekar <anish.shrigonde...@databricks.com.invalid> wrote:
> +1 - Would be great to get this into the Spark 4.0 release. > > Thanks, > Anish > > On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Hi dev, >> >> We are going to introduce a new API named `transformWithState` for >> streaming query, which allows users to perform more complex stateful >> operation in user function, with lot simpler code compared to >> `flatMapGroupsWithState` (and `applyInPandasWithState`). >> >> The target version has been Spark 4.0.0 and we track this project as a >> major one for Spark 4. We push most planned features into Spark 4.0.0, >> except Spark Connect support. >> >> The PRs for Spark Connect support are merged into Spark 4.1 branch, but >> I'm seeking the voice whether we can introduce Spark Connect support to >> Spark 4.0.0. >> >> I understand this arrives a bit late, but since the API is something >> backed by a huge effort and I foresee this new API to replace the usage of >> flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to make >> sure we don't push users back to wait for another 6+ months to use this in >> Spark Connect. >> >> Would love to hear your thoughts. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >
