Thanks for catching this (unfortunately this is already checked in, so changing the PR description doesn't reflect the commit).
But I'd say, let's focus on the discussion and let's not try to nitpick. If we are really concerned, I can modify the commit message when we decide to push this to Spark 4.0.0 to at least correct this in Spark 4.0.0, but this sounds to me as a whole different topic. On Wed, Mar 5, 2025 at 7:02 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks, > > I read PySpark pull. I suggest this > Why are the changes needed? > > As Spark connect is becoming the default *API *in spark 4.0, we need to > add connect support for TWS in Python. > Why: > > Saying this "As Spark Connect is becoming* the default AP*I in Spark 4.0" > reflects more accurately that Spark Connect is an interface for interacting > with Spark, not a replacement for the entire system. > > HTH > .. > > > Dr Mich Talebzadeh, > Architect | Data Science | Financial Crime | Forensic Analysis | GDPR > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > > > On Tue, 4 Mar 2025 at 20:35, Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Hi, >> >> Here are PRs we are seeking for consensus to get in for 4.0. >> >> PySpark: https://github.com/apache/spark/pull/49560 >> Scala: https://github.com/apache/spark/pull/49488 >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> >> On Tue, Mar 4, 2025 at 11:06 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Thanks. >>> >>> Can you point to a link or any further documentation please? >>> >>> Dr Mich Talebzadeh, >>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> >>> >>> >>> On Tue, 4 Mar 2025 at 13:22, Herman van Hovell >>> <her...@databricks.com.invalid> wrote: >>> >>>> +1 >>>> >>>> On Tue, Mar 4, 2025 at 2:07 AM Anish Shrigondekar >>>> <anish.shrigonde...@databricks.com.invalid> wrote: >>>> >>>>> +1 - Would be great to get this into the Spark 4.0 release. >>>>> >>>>> Thanks, >>>>> Anish >>>>> >>>>> On Mon, Mar 3, 2025 at 9:35 PM Jungtaek Lim < >>>>> kabhwan.opensou...@gmail.com> wrote: >>>>> >>>>>> Hi dev, >>>>>> >>>>>> We are going to introduce a new API named `transformWithState` for >>>>>> streaming query, which allows users to perform more complex stateful >>>>>> operation in user function, with lot simpler code compared to >>>>>> `flatMapGroupsWithState` (and `applyInPandasWithState`). >>>>>> >>>>>> The target version has been Spark 4.0.0 and we track this project as >>>>>> a major one for Spark 4. We push most planned features into Spark 4.0.0, >>>>>> except Spark Connect support. >>>>>> >>>>>> The PRs for Spark Connect support are merged into Spark 4.1 branch, >>>>>> but I'm seeking the voice whether we can introduce Spark Connect support >>>>>> to >>>>>> Spark 4.0.0. >>>>>> >>>>>> I understand this arrives a bit late, but since the API is something >>>>>> backed by a huge effort and I foresee this new API to replace the usage >>>>>> of >>>>>> flatMapGroupsWithState and applyInPandasWithState sooner, I'd like to >>>>>> make >>>>>> sure we don't push users back to wait for another 6+ months to use this >>>>>> in >>>>>> Spark Connect. >>>>>> >>>>>> Would love to hear your thoughts. >>>>>> >>>>>> Thanks, >>>>>> Jungtaek Lim (HeartSaVioR) >>>>>> >>>>>
