> Isn't it a bit excessive to talk about "making the voice of the community very clear" when only a very small fraction of Spark users have participated in the discussion or cast their votes?
This has been an ongoing issue for every single project. You get less votes if you initiate VOTE for more sensitive topics. If this were just about whether to ship bugfix release, I think it's trivial to get more than 5 binding votes in a couple days. For this VOTE, it's somehow interpreted as a sensitive and political one (though it is never intended), hence it was really uneasy to just get 3 +1 binding votes. For this topic, I think the most important opinion is from users, but there is no way we can hear from a meaningful number of users, like 1000 users. I think the VOTE includes users' +1 who are contributors but also maintain streaming queries in their daily job. This is all about the best effort we can make realistically. Back to your vote, again, the user facing configuration will be removed in Spark 4.0.0. This is unchanged. You don't have a voting position yet, because you don't seem to understand the proposal properly. The thing we are talking about is, the migration logic needs to refer to the old config name (having vendor name), since we need to rewrite the value of old config to new config for offset log to fix it and move further. This is needed "only once for each streaming query". Due to the rewrite logic, it is unavoidable to refer to the vendor name as "string" if we want to retain the migration logic. And there are few objections about even having a vendor name in a string. That's that. Simply speaking, there are conflicting proposals: 1) kick out the migration logic and enforce users about their upgrade path to be controlled by us (telling users who use Spark 3.5.4 to upgrade to Spark 3.5.5 first to upgrade to Spark 4.0), to remove the vendor name ASAP 2) retain the migration logic to allow some time frame of upgradability which unfortunately carries the vendor name a bit longer. and current VOTE is to move forward with 2). We now have a VOTE thread initiated from Mark, and from his post on VOTE I think he also understands what the proposal is, so you can read his content on VOTE if you need more clarification. On Sun, Mar 16, 2025 at 2:19 PM Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com> wrote: > Isn't it a bit excessive to talk about "making the voice of the community > very clear" when only a very small fraction of Spark users have > participated in the discussion or cast their votes? > > That said, I read the initial emails and understood the proposal. Since it > seemed obvious and straightforward to me, I didn’t pay much attention to > the ongoing discussion. However, after this long thread, I’m now unclear > about the issue—why is there hesitation in dropping a configuration that > was never supposed to be there in the first place, especially in a major > release like Spark 4? > > Rather than going through the entire email chain, I’d appreciate a quick > confirmation: I support keeping it as is in Spark 3.5 and removing it in > Spark 4. Would that be considered a -1 or a +1? (non-binding.) > > Thanks, > > El dom, 16 mar 2025, 3:36, Jungtaek Lim <kabhwan.opensou...@gmail.com> > escribió: > >> Thanks Mark for trying to make the voice from community very clear and be >> recorded. Now community is required to just validate Dongjoon’s veto based >> on his last post and there is no longer a concern about vote process with >> your proposal. >> If we agree that Dongjoon’s justification is moot, I’ll need to update >> the result to NOT contain Dongjoon’s -1, as it is not justified, and it’s >> passed with no veto even if we consider the prior VOTE to be code change. >> >> I agree this is lot clearer way for us to push community to express their >> voice very seriously, including their justification super clearly, and >> stand on “one side” rather than trying to nitpick the process to block it. >> Thanks again Mark, this is awesome. >> >> 2025년 3월 16일 (일) 오전 8:36, Mark Hamstra <markhams...@gmail.com>님이 작성: >> >>> I don’t think we quite need to involve the board yet. There is still the >>> open question of whether the technical justification for the veto is valid, >>> and it is purely up to the PMC to resolve those doubts. >>> >>> I don’t understand the apparent reluctance to call for resolution of >>> that question even though the technical justification (and thus the veto) >>> has been challenged in comments, so I’ll do it myself momentarily…. >>> >>> >>> On Sat, Mar 15, 2025 at 2:33 PM Holden Karau <holden.ka...@gmail.com> >>> wrote: >>> >>>> My $0.02: >>>> >>>> I do not believe that this vote has passed. I believe there is a valid >>>> veto. On a personal level from a migration point of view I think Spark 4 is >>>> the perfect time to drop this configuration. >>>> >>>> Given the disagreement of if this is a valid veto I think we should >>>> pause until the board has been given time to provide its input. >>>> >>>> Previous perceived dismissing of committer and PMC input has resulted >>>> in this being a more sensitive topic in this project than perhaps some >>>> others. >>>> >>>> How do folks feel about writing up their views for the board sending it >>>> over to them and seeing if they want to provide input? >>>> >>>> Twitter: https://twitter.com/holdenkarau >>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> Pronouns: she/her >>>> >>>> >>>> On Sat, Mar 15, 2025 at 1:38 PM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> This is my gist >>>>> >>>>> Mark from your passionate language I gather you see this as a "Code >>>>> Change" veto. Your reasoning seems to be straightforward, i.e. the vote's >>>>> purpose is to decide whether to add code (migration logic) to the Spark >>>>> 4.0 >>>>> branch. In your view, the outcome of the vote directly alters the >>>>> software's code? >>>>> >>>>> However, If we see it as a *procedural matter only* like some others >>>>> include myself, it involves a vote and the interpretation of rules. >>>>> >>>>> In summary >>>>> >>>>> 1. If it is a code change vote, a -1 can be seen as a veto, >>>>> blocking the change unless specific conditions are met (like the PMC >>>>> overriding the veto). >>>>> 2. If it is just a procedural vote, a -1 might simply be a >>>>> dissenting vote, *not necessarily carrying the power to block the >>>>> entire action.* >>>>> >>>>> FYI, I recall I voted -1 (non binding) on another thread and >>>>> Dongioon asked me to explain which it was in his right >>>>> >>>>> I can see the following vote cast (1) >>>>> >>>>> - Jungtaek Lim: +1 (non-binding) >>>>> - Sean Owen: +1 to retain >>>>> - Yang Jie: -1, later withdraws it and casts +1 >>>>> - Adam Binford: +1 (non-binding) >>>>> - Russell Jurney: +1 non-binding >>>>> - Yang Jie: +1 >>>>> - Mridul Muralidharan: +1 >>>>> - Dongjoon Hyun: -1 >>>>> >>>>> >>>>> *(1) Summary of Voting from 21 emails in the attached file* from >>>>> https://lists.apache.org/thread/nm3p1zjcybdl0p0mc56t2rl92hb9837n : >>>>> >>>>> For Retaining Migration Logic (+1): Jungtaek Lim, Sean Owen, Yang Jie >>>>> (initially -1, then +1), Adam Binford (non-binding), Russell Jurney >>>>> (non-binding), Mridul Muralidharan >>>>> Against Retaining Migration Logic (-1): Dongjoon Hyun >>>>> >>>>> Maybe we should put a bar on it and allow Dongjoon to qualify his >>>>> statement as 1 or 2 above, thern it could be escalated if needed or put at >>>>> rest >>>>> >>>>> HTH >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sat, 15 Mar 2025 at 18:26, Mark Hamstra <markhams...@gmail.com> >>>>> wrote: >>>>> >>>>>> That is utter nonsense, Sean! You do not have any authority to declare >>>>>> the matter concluded, and I will escalate to the board if you persist >>>>>> in this approach. >>>>>> >>>>>> The proposed code change has been vetoed. As I delineated previously, >>>>>> there are two and only two ways forward under the ASF Voting Process. >>>>>> That does not include any individual simply declaring that the matter >>>>>> has been concluded regardless of the veto and ASF process. >>>>>> >>>>>> On Sat, Mar 15, 2025 at 11:18 AM Sean Owen <sro...@gmail.com> wrote: >>>>>> > >>>>>> > Mark et al - this thread has gone on way too long. Everyone has >>>>>> expressed their opinion. The result stands. >>>>>> > Anyone who is really upset about it, please escalate to the board >>>>>> or something, but, this thread and decision point has now concluded. >>>>>> > >>>>>> > >>>>>> > On Sat, Mar 15, 2025 at 1:16 PM Mark Hamstra <markhams...@gmail.com> >>>>>> wrote: >>>>>> >> >>>>>> >> You do not have the authority to declare Dongjoon's technical >>>>>> >> justification invalid. That is up to the PMC: "In case of doubt, >>>>>> >> deciding whether a technical justification is valid is up to the >>>>>> PMC." >>>>>> >> >>>>>> >> On Sat, Mar 15, 2025 at 6:20 AM Jungtaek Lim >>>>>> >> <kabhwan.opensou...@gmail.com> wrote: >>>>>> >> > >>>>>> >> > To summarize, the main arguments of both proposals are "whether >>>>>> we can force users to upgrade to Spark 3.5.5 first before upgrading Spark >>>>>> 4.0.0" vs "we should include migration logic to Spark 4.0.0 because that >>>>>> is >>>>>> not realistic". Where is the "technical objection" here? If you say there >>>>>> was politics I can clearly say never, but even if you interpret there was >>>>>> politics, politics is not "technical objection". I can quote the relevant >>>>>> ASF page for you. >>>>>> >> > >>>>>> >> > https://www.apache.org/foundation/voting.html#Veto >>>>>> >> > >>>>>> >> > > A -1 vote by a qualified voter stops a code-modification >>>>>> proposal in its tracks. This constitutes a veto, and it cannot be >>>>>> overruled >>>>>> nor overridden by anyone. Vetoes stand until and unless the individual >>>>>> withdraws their veto. >>>>>> >> > > To prevent vetoes from being used capriciously, the voter must >>>>>> provide with the veto a technical justification showing why the change is >>>>>> bad (opens a security exposure, negatively affects performance, etc. ). A >>>>>> veto without a justification is invalid and has no weight. >>>>>> >> > >>>>>> >> > The justification must be "technical one" for vote. I hope ASF >>>>>> just lists the most cases rather than leaving this as etc, but I think >>>>>> ASF >>>>>> believes individual's judgement, and I claim there is no "technical >>>>>> reason". Having to put 4 more lines is never a technical reason. It is >>>>>> never meant to be used for blocking different opinions. It must be used >>>>>> for >>>>>> blocking "incidents which impact users", while we are here to do the >>>>>> opposite, saving users' life. >>>>>> >> > >>>>>> >> > On Sat, Mar 15, 2025 at 9:51 PM Jungtaek Lim < >>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>> >> >> >>>>>> >> >> > That's the reason why you proposed the vote procedure and we >>>>>> agreed. >>>>>> >> >> >>>>>> >> >> Didn’t you see the part “we agreed”? Who is we in the context? >>>>>> >> >> >>>>>> >> >> I don’t think he answered my questions - he explained his >>>>>> reasoning of his proposal which majorly does not agree with. You even >>>>>> said >>>>>> uou are not persuaded and I want to ask you now you were persuaded from >>>>>> his >>>>>> last post. >>>>>> >> >> >>>>>> >> >> Again I haven’t heard my answers. He showed his reasoning but >>>>>> there is nothing about the evidence of the validity of “technical” >>>>>> objection. I think I have asked people who judged his -1 as veto for >>>>>> their >>>>>> reasoning of how this could be “technical” objection and I don’t think I >>>>>> heard anything. >>>>>> >> >> >>>>>> >> >> I can be corrected if you can point out what is the “technical” >>>>>> objection. If you or Dongjoon do not provide this to the end of the >>>>>> week, I >>>>>> have to consider I haven’t heard about that and the veto (although >>>>>> Dongjoon >>>>>> stated it is not a veto) will be ignored. >>>>>> >> >> >>>>>> >> >> 2025년 3월 15일 (토) 오후 8:19, Mark Hamstra <markhams...@gmail.com>님이 >>>>>> 작성: >>>>>> >> >>> >>>>>> >> >>> Once again, I have to object. Dongjoon said that the vote is a >>>>>> time >>>>>> >> >>> limited procedure, not that the vote itself is a procedural >>>>>> vote as >>>>>> >> >>> distinct from a code change vote or a package release vote. >>>>>> >> >>> >>>>>> >> >>> Frankly, this feels like you are trying to manipulate the vote >>>>>> >> >>> procedure by misrepresenting Dongjoon, and you are quickly >>>>>> losing my >>>>>> >> >>> confidence in your ability to administer a fair voting >>>>>> procedure. >>>>>> >> >>> >>>>>> >> >>> I still consider the proposal to be vetoed. >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> On Fri, Mar 14, 2025 at 6:11 PM Jungtaek Lim >>>>>> >> >>> <kabhwan.opensou...@gmail.com> wrote: >>>>>> >> >>> > >>>>>> >> >>> > UPDATE: >>>>>> >> >>> > >>>>>> >> >>> > We were having a discussion about the type of VOTE, since >>>>>> Dongjoon's -1 should be considered as a veto if we see this as a code >>>>>> change VOTE. >>>>>> >> >>> > Dongjoon clarified that he does not see this VOTE as a code >>>>>> change, hence he gave -1 but not intended to block the VOTE. >>>>>> >> >>> > >>>>>> >> >>> > That said, we have confirmed that Dongjoon's -1 is not a >>>>>> veto. I think the VOTE result is correct as it is. I'll proceed with the >>>>>> next steps. >>>>>> >> >>> > >>>>>> >> >>> > On Fri, Mar 14, 2025 at 11:19 AM Jungtaek Lim < >>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>> >> >>> >> >>>>>> >> >>> >> The vote passes with 7 +1s (3 binding +1s) and 1 -1s (1 >>>>>> binding -1s). >>>>>> >> >>> >> Thanks to all who helped with the vote! >>>>>> >> >>> >> >>>>>> >> >>> >> I'm going to make a code change in branch-4.0 quickly so >>>>>> that we don't have to trigger another RC for Spark 4.0.0 just because of >>>>>> this. >>>>>> >> >>> >> >>>>>> >> >>> >> (* = binding) >>>>>> >> >>> >> +1: >>>>>> >> >>> >> - Sean R. Owen * >>>>>> >> >>> >> - Jungtaek Lim >>>>>> >> >>> >> - Nicholas Chammas >>>>>> >> >>> >> - Wenchen Fan * >>>>>> >> >>> >> - Adam Binford >>>>>> >> >>> >> - Russell Jurney >>>>>> >> >>> >> - Yang Jie * >>>>>> >> >>> >> >>>>>> >> >>> >> -1: >>>>>> >> >>> >> - Dongjoon Hyun * >>>>>> >> >>> >> >>>>>> >> >>> >> Thanks, >>>>>> >> >>> >> Jungtaek Lim (HeartSaVioR) >>>>>> >> >>> >>>>>> >> >>> >>>>>> --------------------------------------------------------------------- >>>>>> >> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>> >> >>> >>>>>> >> >>>>>> >> >>>>>> --------------------------------------------------------------------- >>>>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>> >> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>>
