Hi Dongjoon, Thanks a lot for your detailed feedback and great questions! Let me clarify my current proposal and thoughts:
1. Regarding Spark 5.0 schedule At the moment, I don’t have a concrete Spark 5.0 schedule in mind. I included the stable major releases in the Final Success criteria mainly to set a practical milestone to complete the automation work and fully transition to automated official releases. I don't intend to set the next major release timeline in this SPIP. 2. Lowering the bar for preview releases In short, yeah. I expect the bar for preview releases to be lower compared to official releases, given that these previews are primarily for early testing and feedback. That said, if the community raises concerns during the vote and we end up with multiple RCs, that’s totally fine. In such cases, we could even skip the next month's preview if needed. My intention is not to strictly enforce monthly previews but to provide regular opportunities for testing, while keeping the process low-pressure for the community. 3. Scope of monthly previews (first minor versions only) Yup. This proposal is only for previews of the next minor version from the master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until we cut the real 4.1.0 release. Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on. There will be no 4.0.1-preview1 style releases under this proposal. 4. Official releases (e.g., 4.0.1, 3.5.7) For now, this SPIP does not target automating or introducing monthly maintenance releases like 4.0.1. But yeah, that's my final goal actually. The automated maintenance releases are where I want to go next, after proving the automation works reliably via previews. here are actually some more work to be done to make it actually no manual step at all. *TL;DR*: this is a step before automating the official releases (it's not tied to the official releases yet to be conservative) + providing users with early access to the latest dev Spark build. On Wed, 2 Jul 2025 at 09:27, DB Tsai <dbt...@dbtsai.com> wrote: > Thank you, Hyukjin for driving the SPIP and for your work on the release > automation infrastructure — it’s a huge step forward. > > I’ve been thinking about this topic quite a bit since the Spark 4.0 > release. While Spark continues to deliver meaningful improvements in every > release and enjoys active community contributions, there’s a lingering > perception that the project is mature but not evolving quickly. I feel this > perception is largely due to the long gap between major versions — it’s > been five years between Spark 3.0 and 4.0 — which has understandably caused > some frustration among both contributors and users. > > Now, with release automation and monthly preview builds purposed in this > SPIP, we have a real opportunity to change that. As Dongjoon suggested, > setting up a regular maintenance release cadence — perhaps bi-monthly > instead of monthly — could strike the right balance and make these builds > more viable for production environments. > > If this model proves successful, we could move toward an even faster major > release cadence and designate one LTS versions annually, with extended > backport support. > > Benefits for the OSS Community: > > - Faster time-to-production for new features > - Stronger contributor engagement > - Quicker community feedback cycles > - Easier debugging and testing through smaller, incremental changes > > > Thanks, > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <dongj...@apache.org> wrote: > > Thank you so much for the suggestion and achieving the automated infra, > Hyujkin. > > I have a few questions. > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as > "Q8. Final Success" criteria. I'm wondering if you have some schedule in > your mind for Spark 5.0 in next 2 years? > > 2. Are we going to lower the bar for the monthly preview releases? > Specifically, I'm wondering if the Preview-RC1 supposed to pass always > because it's a preview release? As we know, it's not until now. For > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK > 4.0.0-preview1 (RC3)". > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is > no `4.0.1-preview1`? > > 4. Although there was an automated test email for 3.5.7, SPIP is not > aiming maintenance version release like 4.0.1 and 3.5.7? > > Initially, I thought you were going to propose (4) "Automated Monthly > Maintenance Release". > > For me, (4) is more beneficial than this SPIP because > - We spend the same community effort during voting (including at least 3 > PMC votes) for (4) and SPIP. > - (4) has the real benefit because users can use it in the production > while SPIP didn't. > > Technically, it could be a little weird if Apache Spark community releases > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual > maintenance versions like `4.0.1`, ..., `4.0.2`. > > In short, "Automated Monthly Maintenance Release" might be the > prerequisite for "Monthly Preview Release". What do you think about that? > Can we extend your SPIP in this direction? > > Thanks, > Dongjoon. > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote: > > Hi all, > > I would like to propose a monthly preview for our dev branch, e.g., Spark > 4.1.0 preview1 ... previewN. > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized > the manual work so I think it's realistic to propose this. > > Couple of notes: > - The manual steps it requires would be to run GitHub Actions twice for RC > and publishing, and summarizing the vote result. There IS a way to even > automate this but it needs more work to comply with ASF policy. I would > like to stick to this minimal manual work for now. > - For now, I would like to volunteer to be responsible for the preview > releases and incrementally improve our release policy guidelines ( > https://spark.apache.org/release-process.html) as well once this SPIP > passes. > - The individual release would be, I suspect, about the first week in each > month but I would like to avoid setting the explicit date in the SPIP so it > makes us less pressured. > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 > SPIP: > > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > >
