Thank you, Hyukjin for driving the SPIP and for your work on the release automation infrastructure — it’s a huge step forward.
I’ve been thinking about this topic quite a bit since the Spark 4.0 release. While Spark continues to deliver meaningful improvements in every release and enjoys active community contributions, there’s a lingering perception that the project is mature but not evolving quickly. I feel this perception is largely due to the long gap between major versions — it’s been five years between Spark 3.0 and 4.0 — which has understandably caused some frustration among both contributors and users. Now, with release automation and monthly preview builds purposed in this SPIP, we have a real opportunity to change that. As Dongjoon suggested, setting up a regular maintenance release cadence — perhaps bi-monthly instead of monthly — could strike the right balance and make these builds more viable for production environments. If this model proves successful, we could move toward an even faster major release cadence and designate one LTS versions annually, with extended backport support. Benefits for the OSS Community: Faster time-to-production for new features Stronger contributor engagement Quicker community feedback cycles Easier debugging and testing through smaller, incremental changes Thanks, DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <dongj...@apache.org> wrote: > > Thank you so much for the suggestion and achieving the automated infra, > Hyujkin. > > I have a few questions. > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major releases") as "Q8. > Final Success" criteria. I'm wondering if you have some schedule in your mind > for Spark 5.0 in next 2 years? > > 2. Are we going to lower the bar for the monthly preview releases? > Specifically, I'm wondering if the Preview-RC1 supposed to pass always > because it's a preview release? As we know, it's not until now. For example, > we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK 4.0.0-preview1 > (RC3)". > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR versions like > Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? There is no > `4.0.1-preview1`? > > 4. Although there was an automated test email for 3.5.7, SPIP is not aiming > maintenance version release like 4.0.1 and 3.5.7? > > Initially, I thought you were going to propose (4) "Automated Monthly > Maintenance Release". > > For me, (4) is more beneficial than this SPIP because > - We spend the same community effort during voting (including at least 3 PMC > votes) for (4) and SPIP. > - (4) has the real benefit because users can use it in the production while > SPIP didn't. > > Technically, it could be a little weird if Apache Spark community releases > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering the actual > maintenance versions like `4.0.1`, ..., `4.0.2`. > > In short, "Automated Monthly Maintenance Release" might be the prerequisite > for "Monthly Preview Release". What do you think about that? Can we extend > your SPIP in this direction? > > Thanks, > Dongjoon. > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote: >> Hi all, >> >> I would like to propose a monthly preview for our dev branch, e.g., Spark >> 4.1.0 preview1 ... previewN. >> >> Per https://issues.apache.org/jira/browse/SPARK-52176, we have minimized >> the manual work so I think it's realistic to propose this. >> >> Couple of notes: >> - The manual steps it requires would be to run GitHub Actions twice for RC >> and publishing, and summarizing the vote result. There IS a way to even >> automate this but it needs more work to comply with ASF policy. I would >> like to stick to this minimal manual work for now. >> - For now, I would like to volunteer to be responsible for the preview >> releases and incrementally improve our release policy guidelines ( >> https://spark.apache.org/release-process.html) as well once this SPIP >> passes. >> - The individual release would be, I suspect, about the first week in each >> month but I would like to avoid setting the explicit date in the SPIP so it >> makes us less pressured. >> >> JIRA: https://issues.apache.org/jira/browse/SPARK-52625 >> SPIP: >> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 >> > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >
