Re: [DISCUSS] SPIP: Monthly preview release
Great point. I would eventually like to remove the vote overhead as well. It is required per ASF policy but I am actually taking a look around to work around this. I took it as monthly because it seems to be pretty enough given my experience of releasing Spark 3.5.6 via automated GitHub Actions. It's more to try it out and see how it goes actually so I think we won't have to worry too much. I also don't want us to strictly force the monthly release - we could skip a couple if there's anything happening. If things don't go well, we can always change / adjust timelines, etc. On Wed, 2 Jul 2025 at 20:39, Mark Hamstra wrote: > On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon wrote: > > > > > 1. Since we "release" the preview, we will go through the VOTE > process. What is the expected overhead on doing this monthly? (Maybe this > would be coupled with the questions below.) > > > > We have to vote for every preview every month for now. We can try and > see how it goes. > > The voting requirement does make me question the overhead as well. I'm > not sure that we can expect the number of PMC members necessary for a > valid vote to give a meaningful release review every month. That would > leave us with automated preview releases with invalid or failed > release votes, or perhaps even worse, preview-releases that have been > rubber-stamped without meaningful review. Beyond naming/accounting > details such as whether the next preview release after preview-n fails > to pass the vote threshold is a potentially confusing repeat of > preview-n or automatically becomes preview-n+1 or preview-n+0.1 or > something, a build up of failed, non-released "releases" would seem to > create its own uncertainty, friction and overhead. > > Given arguendo that the span between actual releases is currently too > long and that the other extreme of an automated release after every > commit would be too short, what is the basis for monthly instead of > maybe something like bi-monthly or quarterly being the right > preview-release cadence? > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: [DISCUSS] SPIP: Monthly preview release
IMHO it's probably better to avoid trying to figure out a workaround if that is the "policy" of ASF. Btw, I guess the main point here is, what is the definition of "acceptable" release. For example, do we feel overloaded to just verify the signature per month? I think we wouldn't. The reason the topic of overhead on the voting came up is because we are not very clear about what the shape of release we want to have for previews. IIUC about SPIP (also answers from Hyukjin w.r.t. my questions), the demand of having preview releases is closer to the demand we want to "tag" the snapshot at some point and release artifacts so that people (especially early adopters and 3rd parties) do not need to struggle with how to pull snapshot artifacts, or even have to build on their own. If that is the case and we are happy with that direction, maybe the fact CI passes would be enough to give +1 to the release along with signature verification. If we want to provide specific meaning to the preview release (e.g. we release preview for feature A being completed), it's probably non-trivial to set the timeline in prior and it has to be flexible rather than specific interval. But if we are on the same page that it's really a pain to play with a snapshot and we want to provide some checkpointed artifacts in the middle of development, I think this is worth giving a try. At least it is not a one way door and interval is not a strict one which can change later upon more lightweight discussion. On Thu, Jul 3, 2025 at 8:59 AM Hyukjin Kwon wrote: > Great point. I would eventually like to remove the vote overhead as well. > It is required per ASF policy but I am actually taking a look around to > work around this. > I took it as monthly because it seems to be pretty enough given my > experience of releasing Spark 3.5.6 via automated GitHub Actions. > > It's more to try it out and see how it goes actually so I think we won't > have to worry too much. > I also don't want us to strictly force the monthly release - we could skip > a couple if there's anything happening. > If things don't go well, we can always change / adjust timelines, etc. > > > On Wed, 2 Jul 2025 at 20:39, Mark Hamstra wrote: > >> On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon wrote: >> > >> > > 1. Since we "release" the preview, we will go through the VOTE >> process. What is the expected overhead on doing this monthly? (Maybe this >> would be coupled with the questions below.) >> > >> > We have to vote for every preview every month for now. We can try and >> see how it goes. >> >> The voting requirement does make me question the overhead as well. I'm >> not sure that we can expect the number of PMC members necessary for a >> valid vote to give a meaningful release review every month. That would >> leave us with automated preview releases with invalid or failed >> release votes, or perhaps even worse, preview-releases that have been >> rubber-stamped without meaningful review. Beyond naming/accounting >> details such as whether the next preview release after preview-n fails >> to pass the vote threshold is a potentially confusing repeat of >> preview-n or automatically becomes preview-n+1 or preview-n+0.1 or >> something, a build up of failed, non-released "releases" would seem to >> create its own uncertainty, friction and overhead. >> >> Given arguendo that the span between actual releases is currently too >> long and that the other extreme of an automated release after every >> commit would be too short, what is the basis for monthly instead of >> maybe something like bi-monthly or quarterly being the right >> preview-release cadence? >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: [VOTE] SPIP: Monthly preview release
+1 On Wed, Jul 2, 2025 at 9:52 PM DB Tsai wrote: > +1 > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > On Jul 2, 2025, at 9:37 PM, Hyukjin Kwon wrote: > > Hi all, > > I would like to start a vote on the monthly preview releases. > > Discussion thread: > https://lists.apache.org/thread/1hmsb3g7lm5k2f9xnp6x2hmss8yrd5h8 > SPIP: > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 > > Please vote on the SPIP for the next 72 hours: > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > > Starting with my own +1. > > >
Re: [VOTE] SPIP: Monthly preview release
+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jul 2, 2025, at 9:37 PM, Hyukjin Kwon wrote: > > Hi all, > > I would like to start a vote on the monthly preview releases. > > Discussion thread: > https://lists.apache.org/thread/1hmsb3g7lm5k2f9xnp6x2hmss8yrd5h8 > SPIP: > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 > > Please vote on the SPIP for the next 72 hours: > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > > Starting with my own +1.
Re: [VOTE] SPIP: Monthly preview release
+1 On Wed, Jul 2, 2025 at 9:38 PM Hyukjin Kwon wrote: > > Hi all, > > I would like to start a vote on the monthly preview releases. > > Discussion thread: > https://lists.apache.org/thread/1hmsb3g7lm5k2f9xnp6x2hmss8yrd5h8 > SPIP: > https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 > JIRA: https://issues.apache.org/jira/browse/SPARK-52625 > > Please vote on the SPIP for the next 72 hours: > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > > Starting with my own +1. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [DISCUSS] SPIP: Monthly preview release
Sounds good. I think this is worth giving a try and we can adjust upon lessons learned if any. Thanks for the proposal! On Wed, Jul 2, 2025 at 1:42 PM Hyukjin Kwon wrote: > > 1. Since we "release" the preview, we will go through the VOTE process. > What is the expected overhead on doing this monthly? (Maybe this would be > coupled with the questions below.) > > We have to vote for every preview every month for now. We can try and see > how it goes. > > > 2. What quality do we foresee on the preview release? Do we expect to > have -1 and address them per monthly release, if there is an unresolved > correctness issue or regression? Or will we have very loose criteria of > verification and leave the preview to early adopters with risks? Is the > stability of preview release based on the fact existing unit tests are > passing, and we are mostly verifying the signatures of artifacts? > > I actually expect almost no -1 unless the release is completely useless > (e.g., can't run anything or can't run Spark submission). But I would > expect the community to decide this :-). > > > 3. Do we allow breaking changes across previews, say, as long as we > don't introduce breaking changes in the new minor version it should be fine? > > Yeah. It's not an official release. It should be fine to break. > > > 4. Does this drive the change of our development > in other directions e.g. dev branch or so? e.g. transformWithState had been > marked as private at the phase of development since we knew it would take a > lot of time. Or will we allow preview releases to contain incomplete > features? How do we prevent users from accessing the incomplete feature, or > do we intend to try it out while it's still in development? > > We won't apply any compatibility policies here I believe. Should be > treated as the same as our past preview releases which does not guarantee > anything. > > > On Wed, 2 Jul 2025 at 11:54, Jungtaek Lim > wrote: > >> Thanks for the proposal. The direction is awesome - we have such a long >> interval between minor releases and this would help us address some of the >> issues from the long release cadence. >> >> I'd like to understand a couple of things. >> >> 1. Since we "release" the preview, we will go through the VOTE process. >> What is the expected overhead on doing this monthly? (Maybe this would be >> coupled with the questions below.) >> >> 2. What quality do we foresee on the preview release? Do we expect to >> have -1 and address them per monthly release, if there is an unresolved >> correctness issue or regression? Or will we have very loose criteria of >> verification and leave the preview to early adopters with risks? Is the >> stability of preview release based on the fact existing unit tests are >> passing, and we are mostly verifying the signatures of artifacts? >> >> 3. Do we allow breaking changes across previews, say, as long as we don't >> introduce breaking changes in the new minor version it should be fine? >> >> 4. Does this drive the change of our development in other directions e.g. >> dev branch or so? e.g. transformWithState had been marked as private at the >> phase of development since we knew it would take a lot of time. Or will we >> allow preview releases to contain incomplete features? How do we prevent >> users from accessing the incomplete feature, or do we intend to try it out >> while it's still in development? >> >> >> >> On Wed, Jul 2, 2025 at 11:25 AM Hyukjin Kwon >> wrote: >> >>> Let me start the vote tmr if we're all good with this :-). >>> >>> On Wed, 2 Jul 2025 at 10:12, Anton Okolnychyi >>> wrote: >>> Having monthly preview releases for Spark is going to be huge for projects like Iceberg and Delta. - Anton On Tue, Jul 1, 2025 at 5:43 PM Dongjoon Hyun wrote: > Thank you for the clarification, Hyukjin. Also, thank you for sharing > your direction, DB. > > I agree with you folks that the AS-IS scope of SPIP is a good start. > > +1 for the SPIP because `4.1.0-previewX` itself is actually very > helpful already during developing Spark subprojects like "Spark Connect > for > Swift" and "Spark K8s Operator". :-) > > Thank you again. > > Dongjoon. > > On 2025/07/02 00:31:10 Hyukjin Kwon wrote: > > Hi Dongjoon, > > > > Thanks a lot for your detailed feedback and great questions! > > Let me clarify my current proposal and thoughts: > > > > 1. Regarding Spark 5.0 schedule > > At the moment, I don’t have a concrete Spark 5.0 schedule in mind. > > I included the stable major releases in the Final Success criteria > mainly > > to set a practical milestone to complete the automation work and > fully > > transition to automated official releases. > > I don't intend to set the next major release timeline in this SPIP. > > > > 2. Lowering the bar for preview releases > > In short, yeah. I expect the bar for preview
Re: [DISCUSS] SPIP: Monthly preview release
On Tue, Jul 1, 2025 at 9:42 PM Hyukjin Kwon wrote: > > > 1. Since we "release" the preview, we will go through the VOTE process. > > What is the expected overhead on doing this monthly? (Maybe this would be > > coupled with the questions below.) > > We have to vote for every preview every month for now. We can try and see how > it goes. The voting requirement does make me question the overhead as well. I'm not sure that we can expect the number of PMC members necessary for a valid vote to give a meaningful release review every month. That would leave us with automated preview releases with invalid or failed release votes, or perhaps even worse, preview-releases that have been rubber-stamped without meaningful review. Beyond naming/accounting details such as whether the next preview release after preview-n fails to pass the vote threshold is a potentially confusing repeat of preview-n or automatically becomes preview-n+1 or preview-n+0.1 or something, a build up of failed, non-released "releases" would seem to create its own uncertainty, friction and overhead. Given arguendo that the span between actual releases is currently too long and that the other extreme of an automated release after every commit would be too short, what is the basis for monthly instead of maybe something like bi-monthly or quarterly being the right preview-release cadence? - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
[VOTE] SPIP: Monthly preview release
Hi all, I would like to start a vote on the monthly preview releases. Discussion thread: https://lists.apache.org/thread/1hmsb3g7lm5k2f9xnp6x2hmss8yrd5h8 SPIP: https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67 JIRA: https://issues.apache.org/jira/browse/SPARK-52625 Please vote on the SPIP for the next 72 hours: [ ] +1: Accept the proposal as an official SPIP [ ] +0 [ ] -1: I don’t think this is a good idea because … Starting with my own +1.
