Let me start the vote tmr if we're all good with this :-). On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <aokolnyc...@gmail.com> wrote:
> Having monthly preview releases for Spark is going to be huge for projects > like Iceberg and Delta. > > - Anton > > On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <dongj...@apache.org> wrote: > >> Thank you for the clarification, Hyukjin. Also, thank you for sharing >> your direction, DB. >> >> I agree with you folks that the AS-IS scope of SPIP is a good start. >> >> +1 for the SPIP because `4.1.0-previewX` itself is actually very helpful >> already during developing Spark subprojects like "Spark Connect for Swift" >> and "Spark K8s Operator". :-) >> >> Thank you again. >> >> Dongjoon. >> >> On 2025/07/02 00:31:10 Hyukjin Kwon wrote: >> > Hi Dongjoon, >> > >> > Thanks a lot for your detailed feedback and great questions! >> > Let me clarify my current proposal and thoughts: >> > >> > 1. Regarding Spark 5.0 schedule >> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind. >> > I included the stable major releases in the Final Success criteria >> mainly >> > to set a practical milestone to complete the automation work and fully >> > transition to automated official releases. >> > I don't intend to set the next major release timeline in this SPIP. >> > >> > 2. Lowering the bar for preview releases >> > In short, yeah. I expect the bar for preview releases to be lower >> compared >> > to official releases, given that these previews are primarily for early >> > testing and feedback. >> > That said, if the community raises concerns during the vote and we end >> up >> > with multiple RCs, that’s totally fine. In such cases, we could even >> skip >> > the next month's preview if needed. >> > My intention is not to strictly enforce monthly previews but to provide >> > regular opportunities for testing, while keeping the process >> low-pressure >> > for the community. >> > >> > 3. Scope of monthly previews (first minor versions only) >> > Yup. This proposal is only for previews of the next minor version from >> the >> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until >> we >> > cut the real 4.1.0 release. >> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on. >> > There will be no 4.0.1-preview1 style releases under this proposal. >> > >> > 4. Official releases (e.g., 4.0.1, 3.5.7) >> > For now, this SPIP does not target automating or introducing monthly >> > maintenance releases like 4.0.1. >> > But yeah, that's my final goal actually. The automated maintenance >> releases >> > are where I want to go next, after proving the automation works reliably >> > via previews. >> > here are actually some more work to be done to make it actually no >> manual >> > step at all. >> > >> > >> > *TL;DR*: this is a step before automating the official releases (it's >> not >> > tied to the official releases yet to be conservative) + providing users >> > with early access to the latest dev Spark build. >> > >> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <dbt...@dbtsai.com> wrote: >> > >> > > Thank you, Hyukjin for driving the SPIP and for your work on the >> release >> > > automation infrastructure — it’s a huge step forward. >> > > >> > > I’ve been thinking about this topic quite a bit since the Spark 4.0 >> > > release. While Spark continues to deliver meaningful improvements in >> every >> > > release and enjoys active community contributions, there’s a lingering >> > > perception that the project is mature but not evolving quickly. I >> feel this >> > > perception is largely due to the long gap between major versions — >> it’s >> > > been five years between Spark 3.0 and 4.0 — which has understandably >> caused >> > > some frustration among both contributors and users. >> > > >> > > Now, with release automation and monthly preview builds purposed in >> this >> > > SPIP, we have a real opportunity to change that. As Dongjoon >> suggested, >> > > setting up a regular maintenance release cadence — perhaps bi-monthly >> > > instead of monthly — could strike the right balance and make these >> builds >> > > more viable for production environments. >> > > >> > > If this model proves successful, we could move toward an even faster >> major >> > > release cadence and designate one LTS versions annually, with extended >> > > backport support. >> > > >> > > Benefits for the OSS Community: >> > > >> > > - Faster time-to-production for new features >> > > - Stronger contributor engagement >> > > - Quicker community feedback cycles >> > > - Easier debugging and testing through smaller, incremental changes >> > > >> > > >> > > Thanks, >> > > >> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> > > >> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <dongj...@apache.org> >> wrote: >> > > >> > > Thank you so much for the suggestion and achieving the automated >> infra, >> > > Hyujkin. >> > > >> > > I have a few questions. >> > > >> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") >> as >> > > "Q8. Final Success" criteria. I'm wondering if you have some schedule >> in >> > > your mind for Spark 5.0 in next 2 years? >> > > >> > > 2. Are we going to lower the bar for the monthly preview releases? >> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass always >> > > because it's a preview release? As we know, it's not until now. For >> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK >> > > 4.0.0-preview1 (RC3)". >> > > >> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR >> versions >> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? >> There is >> > > no `4.0.1-preview1`? >> > > >> > > 4. Although there was an automated test email for 3.5.7, SPIP is not >> > > aiming maintenance version release like 4.0.1 and 3.5.7? >> > > >> > > Initially, I thought you were going to propose (4) "Automated Monthly >> > > Maintenance Release". >> > > >> > > For me, (4) is more beneficial than this SPIP because >> > > - We spend the same community effort during voting (including at >> least 3 >> > > PMC votes) for (4) and SPIP. >> > > - (4) has the real benefit because users can use it in the production >> > > while SPIP didn't. >> > > >> > > Technically, it could be a little weird if Apache Spark community >> releases >> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the >> actual >> > > maintenance versions like `4.0.1`, ..., `4.0.2`. >> > > >> > > In short, "Automated Monthly Maintenance Release" might be the >> > > prerequisite for "Monthly Preview Release". What do you think about >> that? >> > > Can we extend your SPIP in this direction? >> > > >> > > Thanks, >> > > Dongjoon. >> > > >> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote: >> > > >> > > Hi all, >> > > >> > > I would like to propose a monthly preview for our dev branch, e.g., >> Spark >> > > 4.1.0 preview1 ... previewN. >> > > >> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have >> minimized >> > > the manual work so I think it's realistic to propose this. >> > > >> > > Couple of notes: >> > > - The manual steps it requires would be to run GitHub Actions twice >> for RC >> > > and publishing, and summarizing the vote result. There IS a way to >> even >> > > automate this but it needs more work to comply with ASF policy. I >> would >> > > like to stick to this minimal manual work for now. >> > > - For now, I would like to volunteer to be responsible for the preview >> > > releases and incrementally improve our release policy guidelines ( >> > > https://spark.apache.org/release-process.html) as well once this SPIP >> > > passes. >> > > - The individual release would be, I suspect, about the first week in >> each >> > > month but I would like to avoid setting the explicit date in the SPIP >> so it >> > > makes us less pressured. >> > > >> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 >> > > SPIP: >> > > >> > > >> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 >> > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > > >> > > >> > > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
