Sounds good. I think this is worth giving a try and we can adjust upon lessons learned if any. Thanks for the proposal!
On Wed, Jul 2, 2025 at 1:42 PM Hyukjin Kwon <gurwls...@apache.org> wrote: > > 1. Since we "release" the preview, we will go through the VOTE process. > What is the expected overhead on doing this monthly? (Maybe this would be > coupled with the questions below.) > > We have to vote for every preview every month for now. We can try and see > how it goes. > > > 2. What quality do we foresee on the preview release? Do we expect to > have -1 and address them per monthly release, if there is an unresolved > correctness issue or regression? Or will we have very loose criteria of > verification and leave the preview to early adopters with risks? Is the > stability of preview release based on the fact existing unit tests are > passing, and we are mostly verifying the signatures of artifacts? > > I actually expect almost no -1 unless the release is completely useless > (e.g., can't run anything or can't run Spark submission). But I would > expect the community to decide this :-). > > > 3. Do we allow breaking changes across previews, say, as long as we > don't introduce breaking changes in the new minor version it should be fine? > > Yeah. It's not an official release. It should be fine to break. > > > 4. Does this drive the change of our development > in other directions e.g. dev branch or so? e.g. transformWithState had been > marked as private at the phase of development since we knew it would take a > lot of time. Or will we allow preview releases to contain incomplete > features? How do we prevent users from accessing the incomplete feature, or > do we intend to try it out while it's still in development? > > We won't apply any compatibility policies here I believe. Should be > treated as the same as our past preview releases which does not guarantee > anything. > > > On Wed, 2 Jul 2025 at 11:54, Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Thanks for the proposal. The direction is awesome - we have such a long >> interval between minor releases and this would help us address some of the >> issues from the long release cadence. >> >> I'd like to understand a couple of things. >> >> 1. Since we "release" the preview, we will go through the VOTE process. >> What is the expected overhead on doing this monthly? (Maybe this would be >> coupled with the questions below.) >> >> 2. What quality do we foresee on the preview release? Do we expect to >> have -1 and address them per monthly release, if there is an unresolved >> correctness issue or regression? Or will we have very loose criteria of >> verification and leave the preview to early adopters with risks? Is the >> stability of preview release based on the fact existing unit tests are >> passing, and we are mostly verifying the signatures of artifacts? >> >> 3. Do we allow breaking changes across previews, say, as long as we don't >> introduce breaking changes in the new minor version it should be fine? >> >> 4. Does this drive the change of our development in other directions e.g. >> dev branch or so? e.g. transformWithState had been marked as private at the >> phase of development since we knew it would take a lot of time. Or will we >> allow preview releases to contain incomplete features? How do we prevent >> users from accessing the incomplete feature, or do we intend to try it out >> while it's still in development? >> >> >> >> On Wed, Jul 2, 2025 at 11:25 AM Hyukjin Kwon <gurwls...@apache.org> >> wrote: >> >>> Let me start the vote tmr if we're all good with this :-). >>> >>> On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi <aokolnyc...@gmail.com> >>> wrote: >>> >>>> Having monthly preview releases for Spark is going to be huge for >>>> projects like Iceberg and Delta. >>>> >>>> - Anton >>>> >>>> On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun <dongj...@apache.org> >>>> wrote: >>>> >>>>> Thank you for the clarification, Hyukjin. Also, thank you for sharing >>>>> your direction, DB. >>>>> >>>>> I agree with you folks that the AS-IS scope of SPIP is a good start. >>>>> >>>>> +1 for the SPIP because `4.1.0-previewX` itself is actually very >>>>> helpful already during developing Spark subprojects like "Spark Connect >>>>> for >>>>> Swift" and "Spark K8s Operator". :-) >>>>> >>>>> Thank you again. >>>>> >>>>> Dongjoon. >>>>> >>>>> On 2025/07/02 00:31:10 Hyukjin Kwon wrote: >>>>> > Hi Dongjoon, >>>>> > >>>>> > Thanks a lot for your detailed feedback and great questions! >>>>> > Let me clarify my current proposal and thoughts: >>>>> > >>>>> > 1. Regarding Spark 5.0 schedule >>>>> > At the moment, I don’t have a concrete Spark 5.0 schedule in mind. >>>>> > I included the stable major releases in the Final Success criteria >>>>> mainly >>>>> > to set a practical milestone to complete the automation work and >>>>> fully >>>>> > transition to automated official releases. >>>>> > I don't intend to set the next major release timeline in this SPIP. >>>>> > >>>>> > 2. Lowering the bar for preview releases >>>>> > In short, yeah. I expect the bar for preview releases to be lower >>>>> compared >>>>> > to official releases, given that these previews are primarily for >>>>> early >>>>> > testing and feedback. >>>>> > That said, if the community raises concerns during the vote and we >>>>> end up >>>>> > with multiple RCs, that’s totally fine. In such cases, we could even >>>>> skip >>>>> > the next month's preview if needed. >>>>> > My intention is not to strictly enforce monthly previews but to >>>>> provide >>>>> > regular opportunities for testing, while keeping the process >>>>> low-pressure >>>>> > for the community. >>>>> > >>>>> > 3. Scope of monthly previews (first minor versions only) >>>>> > Yup. This proposal is only for previews of the next minor version >>>>> from the >>>>> > master branch. For example: 4.1.0-preview1, 4.1.0-preview2, ..., >>>>> until we >>>>> > cut the real 4.1.0 release. >>>>> > Once 4.1.0 is out, previews would move to 4.2.0-preview1, and so on. >>>>> > There will be no 4.0.1-preview1 style releases under this proposal. >>>>> > >>>>> > 4. Official releases (e.g., 4.0.1, 3.5.7) >>>>> > For now, this SPIP does not target automating or introducing monthly >>>>> > maintenance releases like 4.0.1. >>>>> > But yeah, that's my final goal actually. The automated maintenance >>>>> releases >>>>> > are where I want to go next, after proving the automation works >>>>> reliably >>>>> > via previews. >>>>> > here are actually some more work to be done to make it actually no >>>>> manual >>>>> > step at all. >>>>> > >>>>> > >>>>> > *TL;DR*: this is a step before automating the official releases >>>>> (it's not >>>>> > tied to the official releases yet to be conservative) + providing >>>>> users >>>>> > with early access to the latest dev Spark build. >>>>> > >>>>> > On Wed, 2 Jul 2025 at 09:27, DB Tsai <dbt...@dbtsai.com> wrote: >>>>> > >>>>> > > Thank you, Hyukjin for driving the SPIP and for your work on the >>>>> release >>>>> > > automation infrastructure — it’s a huge step forward. >>>>> > > >>>>> > > I’ve been thinking about this topic quite a bit since the Spark 4.0 >>>>> > > release. While Spark continues to deliver meaningful improvements >>>>> in every >>>>> > > release and enjoys active community contributions, there’s a >>>>> lingering >>>>> > > perception that the project is mature but not evolving quickly. I >>>>> feel this >>>>> > > perception is largely due to the long gap between major versions — >>>>> it’s >>>>> > > been five years between Spark 3.0 and 4.0 — which has >>>>> understandably caused >>>>> > > some frustration among both contributors and users. >>>>> > > >>>>> > > Now, with release automation and monthly preview builds purposed >>>>> in this >>>>> > > SPIP, we have a real opportunity to change that. As Dongjoon >>>>> suggested, >>>>> > > setting up a regular maintenance release cadence — perhaps >>>>> bi-monthly >>>>> > > instead of monthly — could strike the right balance and make these >>>>> builds >>>>> > > more viable for production environments. >>>>> > > >>>>> > > If this model proves successful, we could move toward an even >>>>> faster major >>>>> > > release cadence and designate one LTS versions annually, with >>>>> extended >>>>> > > backport support. >>>>> > > >>>>> > > Benefits for the OSS Community: >>>>> > > >>>>> > > - Faster time-to-production for new features >>>>> > > - Stronger contributor engagement >>>>> > > - Quicker community feedback cycles >>>>> > > - Easier debugging and testing through smaller, incremental >>>>> changes >>>>> > > >>>>> > > >>>>> > > Thanks, >>>>> > > >>>>> > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>>>> > > >>>>> > > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun <dongj...@apache.org> >>>>> wrote: >>>>> > > >>>>> > > Thank you so much for the suggestion and achieving the automated >>>>> infra, >>>>> > > Hyujkin. >>>>> > > >>>>> > > I have a few questions. >>>>> > > >>>>> > > 1. Since the SPIP suggests Apache Spark 5.0 ("Stable major >>>>> releases") as >>>>> > > "Q8. Final Success" criteria. I'm wondering if you have some >>>>> schedule in >>>>> > > your mind for Spark 5.0 in next 2 years? >>>>> > > >>>>> > > 2. Are we going to lower the bar for the monthly preview releases? >>>>> > > Specifically, I'm wondering if the Preview-RC1 supposed to pass >>>>> always >>>>> > > because it's a preview release? As we know, it's not until now. For >>>>> > > example, we had three RCs for `4.0.0-preview1` like "[VOTE] SPARK >>>>> > > 4.0.0-preview1 (RC3)". >>>>> > > >>>>> > > 3. Is SPIP proposing monthly previews for only the FIRST MINOR >>>>> versions >>>>> > > like Spark 4.1.0? For example, 4.1.0-preview1 and 4.2.0-preview1? >>>>> There is >>>>> > > no `4.0.1-preview1`? >>>>> > > >>>>> > > 4. Although there was an automated test email for 3.5.7, SPIP is >>>>> not >>>>> > > aiming maintenance version release like 4.0.1 and 3.5.7? >>>>> > > >>>>> > > Initially, I thought you were going to propose (4) "Automated >>>>> Monthly >>>>> > > Maintenance Release". >>>>> > > >>>>> > > For me, (4) is more beneficial than this SPIP because >>>>> > > - We spend the same community effort during voting (including at >>>>> least 3 >>>>> > > PMC votes) for (4) and SPIP. >>>>> > > - (4) has the real benefit because users can use it in the >>>>> production >>>>> > > while SPIP didn't. >>>>> > > >>>>> > > Technically, it could be a little weird if Apache Spark community >>>>> releases >>>>> > > only "4.1.0-preview1", ..., "4.1.0-previewX" without delivering >>>>> the actual >>>>> > > maintenance versions like `4.0.1`, ..., `4.0.2`. >>>>> > > >>>>> > > In short, "Automated Monthly Maintenance Release" might be the >>>>> > > prerequisite for "Monthly Preview Release". What do you think >>>>> about that? >>>>> > > Can we extend your SPIP in this direction? >>>>> > > >>>>> > > Thanks, >>>>> > > Dongjoon. >>>>> > > >>>>> > > On 2025/06/30 23:34:54 Hyukjin Kwon wrote: >>>>> > > >>>>> > > Hi all, >>>>> > > >>>>> > > I would like to propose a monthly preview for our dev branch, >>>>> e.g., Spark >>>>> > > 4.1.0 preview1 ... previewN. >>>>> > > >>>>> > > Per https://issues.apache.org/jira/browse/SPARK-52176, we have >>>>> minimized >>>>> > > the manual work so I think it's realistic to propose this. >>>>> > > >>>>> > > Couple of notes: >>>>> > > - The manual steps it requires would be to run GitHub Actions >>>>> twice for RC >>>>> > > and publishing, and summarizing the vote result. There IS a way to >>>>> even >>>>> > > automate this but it needs more work to comply with ASF policy. I >>>>> would >>>>> > > like to stick to this minimal manual work for now. >>>>> > > - For now, I would like to volunteer to be responsible for the >>>>> preview >>>>> > > releases and incrementally improve our release policy guidelines ( >>>>> > > https://spark.apache.org/release-process.html) as well once this >>>>> SPIP >>>>> > > passes. >>>>> > > - The individual release would be, I suspect, about the first week >>>>> in each >>>>> > > month but I would like to avoid setting the explicit date in the >>>>> SPIP so it >>>>> > > makes us less pressured. >>>>> > > >>>>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 >>>>> > > SPIP: >>>>> > > >>>>> > > >>>>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 >>>>> > > >>>>> > > >>>>> > > >>>>> --------------------------------------------------------------------- >>>>> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> > > >>>>> > > >>>>> > > >>>>> > >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> >>>>>
