We're basically working on making Scala Spark Connect ready. For example, I am working on having a parent class for both Spark Classic and Spark Connect so users would face less breaking changes, and they can run their application without changing anything. In addition, I am also working on sharing the same test base between Spark Classic and Spark Connect. For those, I think it might take a couple of months to stabilize.
What about the below schedule? - 2024-01-15 Creating `branch-4.0` (allowing backporting new features) - 2024-02-01 Feature Freeze (allowing backporting bug fixes only) - 2024-02-15 Starting Apache Spark 4.0.0 RC1 On Fri, 27 Sept 2024 at 07:35, Dongjoon Hyun <dongjoon.h...@gmail.com> wrote: > Thank you for the reply, Herman. > > Given that December and January are on your schedule, > I'm not sure what date your proposal is. Could you elaborate more? > As we know, the community is less active during that Winter period. > > In addition, although I know that you are leading that area actively with > big refactorings, > it would be greatly appreciated if you could share a more concrete > progress status and > delivery plan (or milestone) for `Connect and Classic Scala interface` to > the community. > Specifically, we are curious about how much we achieved between `preview1` > and `preview2`, > and if it's going to be stabilized enough at that time frame. Let's see > what you have more. > > > We are working on unifying the Connect and Classic Scala interface > > Thank you again! > > Dongjoon. > > On Thu, Sep 26, 2024 at 12:23 PM Herman van Hovell <her...@databricks.com> > wrote: > >> Hi, >> >> Can we push back the dates by at least 2 months? >> >> We are working on unifying the Connect and Classic Scala interface, and I >> would like to avoid rushing things. >> >> Kind regards, >> Herman >> >> On Thu, Sep 26, 2024 at 3:19 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> >>> Hi, All. >>> >>> We've delivered two preview releases for Apache Spark 4.0 successfully. >>> I believe it's time to discuss cutting branch-4.0 to stabilize more >>> based on them >>> and schedule feature freeze. >>> >>> - https://spark.apache.org/news/spark-4.0.0-preview1.html (June) >>> - https://spark.apache.org/news/spark-4.0.0-preview2.html (September) >>> >>> I'd like to propose as a candidate. >>> >>> - 2024-10-01 Creating `branch-4.0` (allowing backporting new features) >>> - 2024-10-15 Feature Freeze (allowing backporting bug fixes only) >>> - 2024-11-01 Starting Apache Spark 4.0.0 RC1 >>> >>> WDYT? Please let me know if you have release blockers for Spark 4 or >>> other schedule candidates in your mind. >>> >>> Thanks, >>> Dongjoon. >>> >>
