Having monthly preview releases for Spark is going to be huge for projects like Iceberg and Delta.
- Anton On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <dongj...@apache.org> wrote: > Thank you for the clarification, Hyukjin. Also, thank you for sharing your > direction, DB. > > I agree with you folks that the AS-IS scope of SPIP is a good start. > > +1 for the SPIP because `4.1.0-previewX` itself is actually very helpful > already during developing Spark subprojects like "Spark Connect for Swift" > and "Spark K8s Operator". :-) > > Thank you again. > > Dongjoon. > > On 2025/07/02 00:31:10 Hyukjin Kwon wrote: > > Hi Dongjoon, > > > > Thanks a lot for your detailed feedback and great questions! > > Let me clarify my current proposal and thoughts: > > > > 1. Regarding Spark 5.0 schedule > > At the moment, I don’t have a concrete Spark 5.0 schedule in mind. > > I included the stable major releases in the Final Success criteria mainly > > to set a practical milestone to complete the automation work and fully > > transition to automated official releases. > > I don't intend to set the next major release timeline in this SPIP. > > > > 2. Lowering the bar for preview releases > > In short, yeah. I expect the bar for preview releases to be lower > compared > > to official releases, given that these previews are primarily for early > > testing and feedback. > > That said, if the community raises concerns during the vote and we end up > > with multiple RCs, that’s totally fine. In such cases, we could even skip > > the next month's preview if needed. > > My intention is not to strictly enforce monthly previews but to provide > > regular opportunities for testing, while keeping the process low-pressure > > for the community. > > > > 3. Scope of monthly previews (first minor versions only) > > Yup. This proposal is only for previews of the next minor version from > the > > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., until we > > cut the real 4.1.0 release. > > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on. > > There will be no 4.0.1-preview1 style releases under this proposal. > > > > 4. Official releases (e.g., 4.0.1, 3.5.7) > > For now, this SPIP does not target automating or introducing monthly > > maintenance releases like 4.0.1. > > But yeah, that's my final goal actually. The automated maintenance > releases > > are where I want to go next, after proving the automation works reliably > > via previews. > > here are actually some more work to be done to make it actually no manual > > step at all. > > > > > > *TL;DR*: this is a step before automating the official releases (it's not > > tied to the official releases yet to be conservative) + providing users > > with early access to the latest dev Spark build. > > > > On Wed, 2 Jul 2025 at 09:27, DB Tsai <dbt...@dbtsai.com> wrote: > > > > > Thank you, Hyukjin for driving the SPIP and for your work on the > release > > > automation infrastructure — it’s a huge step forward. > > > > > > I’ve been thinking about this topic quite a bit since the Spark 4.0 > > > release. While Spark continues to deliver meaningful improvements in > every > > > release and enjoys active community contributions, there’s a lingering > > > perception that the project is mature but not evolving quickly. I feel > this > > > perception is largely due to the long gap between major versions — it’s > > > been five years between Spark 3.0 and 4.0 — which has understandably > caused > > > some frustration among both contributors and users. > > > > > > Now, with release automation and monthly preview builds purposed in > this > > > SPIP, we have a real opportunity to change that. As Dongjoon suggested, > > > setting up a regular maintenance release cadence — perhaps bi-monthly > > > instead of monthly — could strike the right balance and make these > builds > > > more viable for production environments. > > > > > > If this model proves successful, we could move toward an even faster > major > > > release cadence and designate one LTS versions annually, with extended > > > backport support. > > > > > > Benefits for the OSS Community: > > > > > > - Faster time-to-production for new features > > > - Stronger contributor engagement > > > - Quicker community feedback cycles > > > - Easier debugging and testing through smaller, incremental changes > > > > > > > > > Thanks, > > > > > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > > > > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <dongj...@apache.org> wrote: > > > > > > Thank you so much for the suggestion and achieving the automated infra, > > > Hyujkin. > > > > > > I have a few questions. > > > > > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") > as > > > "Q8. Final Success" criteria. I'm wondering if you have some schedule > in > > > your mind for Spark 5.0 in next 2 years? > > > > > > 2. Are we going to lower the bar for the monthly preview releases? > > > Specifically, I'm wondering if the Preview-RC1 supposed to pass always > > > because it's a preview release? As we know, it's not until now. For > > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK > > > 4.0.0-preview1 (RC3)". > > > > > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions > > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? > There is > > > no `4.0.1-preview1`? > > > > > > 4. Although there was an automated test email for 3.5.7, SPIP is not > > > aiming maintenance version release like 4.0.1 and 3.5.7? > > > > > > Initially, I thought you were going to propose (4) "Automated Monthly > > > Maintenance Release". > > > > > > For me, (4) is more beneficial than this SPIP because > > > - We spend the same community effort during voting (including at least > 3 > > > PMC votes) for (4) and SPIP. > > > - (4) has the real benefit because users can use it in the production > > > while SPIP didn't. > > > > > > Technically, it could be a little weird if Apache Spark community > releases > > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the > actual > > > maintenance versions like `4.0.1`, ..., `4.0.2`. > > > > > > In short, "Automated Monthly Maintenance Release" might be the > > > prerequisite for "Monthly Preview Release". What do you think about > that? > > > Can we extend your SPIP in this direction? > > > > > > Thanks, > > > Dongjoon. > > > > > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote: > > > > > > Hi all, > > > > > > I would like to propose a monthly preview for our dev branch, e.g., > Spark > > > 4.1.0 preview1 ... previewN. > > > > > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have > minimized > > > the manual work so I think it's realistic to propose this. > > > > > > Couple of notes: > > > - The manual steps it requires would be to run GitHub Actions twice > for RC > > > and publishing, and summarizing the vote result. There IS a way to even > > > automate this but it needs more work to comply with ASF policy. I would > > > like to stick to this minimal manual work for now. > > > - For now, I would like to volunteer to be responsible for the preview > > > releases and incrementally improve our release policy guidelines ( > > > https://spark.apache.org/release-process.html) as well once this SPIP > > > passes. > > > - The individual release would be, I suspect, about the first week in > each > > > month but I would like to avoid setting the explicit date in the SPIP > so it > > > makes us less pressured. > > > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 > > > SPIP: > > > > > > > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
