Hi dev, We are going to introduce a new API named `transformWithState` for streaming query, which allows users to perform more complex stateful operation in user function, with lot simpler code compared to `flatMapGroupsWithState` (and `applyInPandasWithState`).
The target version has been Spark 4.0.0 and we track this project as a major one for Spark 4. We push most planned features into Spark 4.0.0, except Spark Connect support. The PRs for Spark Connect support are merged into Spark 4.1 branch, but I'm seeking the voice whether we can introduce Spark Connect support to Spark 4.0.0. I understand this arrives a bit late, but since the API is something backed by a huge effort and I foresee this new API to replace the usage of flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to make sure we don't push users back to wait for another 6+ months to use this in Spark Connect. Would love to hear your thoughts. Thanks, Jungtaek Lim (HeartSaVioR)
