Btw, I don't think reverting the PR of removing config is violating community's consensus, because we never had a VOTE for this breaking change (this is definitely a breaking change without migration logic), so we did not really get explicit consensus with that. The real neutral state is that we do not have a code change for removing config. (I'm open to hear objections.)
It was merged based on the assumption that we will figure out how to NOT break backward compatibility, and since this is NOT addressed, I argue the change of removing config has to be reverted. Even further, this does not violate the community's consensus since I have strong evidence of VETO on that PR - We generally do not consider breaking change without any alternative to be valid change. I'm happy to go and cast -1 to that PR (committer should have been considered to have a binding vote in code, otherwise my +1 cannot merge the PR), and revert unless anyone objects with valid reason. On Mon, Mar 17, 2025 at 12:31 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Holden, I think I have some workaround like I posted in dev@ (link > <https://lists.apache.org/thread/xsq58800smtc5xo15kfzyj5kfw5yh7wj>), > which is definitely less better than this proposal so I still want to see > this VOTE to go forward, but it's somewhat better in this situation that we > no longer talk about vendor name, hence no need to debate for more minor > versions. > > Though I still want to hear the community's voice that "Are we really > thinking that forcing users to take the upgrade path as we guided is making > sense?". I'm not strongly arguing that the vendor name has to be in the > codebase (but if this is easier way then we should weigh), but I strongly > argue that we have no control of users, so we must always think how we can > achieve letting users do whatever they want, with minimized inconvenience > (since we can't always make things not break). I think this meta question > is a strong argument from me, and I'm -1 to Dongjoon's proposal because his > proposal is breaking this. > > I strongly believe there is NO way to force users to upgrade to a specific > version - I think we should have sought alternatives when this came up with > the main proposal. > > 2025년 3월 17일 (월) 오후 12:10, Holden Karau <holden.ka...@gmail.com>님이 작성: > >> I'm delighted to see folks talking about a compromise. However, instead >> of just asking Dongjoon to withdraw the VETO perhaps folks can suggest >> alternatives that that might meet some of both parties goals? >> >> On Sun, Mar 16, 2025 at 7:41 PM Wenchen Fan <cloud0...@gmail.com> wrote: >> >>> I agree with Holden that withdrawing a veto is always better than >>> overriding it: it's healthier for the community. Dongjoon, would you be >>> willing to reconsider your veto given the current as-is state of the 4.0.0 >>> release (the breaking change will be reverted)? >>> >>> On Mon, Mar 17, 2025 at 10:36 AM Wenchen Fan <cloud0...@gmail.com> >>> wrote: >>> >>>> I've created the revert PR for branch-4.0: >>>> https://github.com/apache/spark/pull/50291 . We can merge PRs with >>>> lazy consensus but it's clear that this breaking change PR has failed to >>>> achieve consensus. >>>> >>>> I hope we now have a clear foundation for discussing solutions. As it >>>> stands, the misnamed configuration will be released in 4.0.0. I like >>>> Jungtaek’s proposal to deprecate it, but the decision is up to the >>>> community. >>>> >>>> On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim < >>>> kabhwan.opensou...@gmail.com> wrote: >>>> >>>>> OK, let's be super honest. >>>>> >>>>> Again, I think you agree that *"both" proposals are "technically" >>>>> correct (or one side can't have a strong theoretical evidence to counter >>>>> the other side)*. So this naturally has a fate to have more >>>>> supporters to get to the end. It's very easy for me to VETO to his >>>>> proposal >>>>> (although I don't have a binding vote, I think I have people who agree >>>>> with >>>>> me) if we think we want to definitely expand the interpretation of VETO >>>>> criteria in the Apache Voting Process. >>>>> >>>>> You said it is up to the PMC member exercising the veto to use their >>>>> judgement, but definitely, it must not be used to force the community to >>>>> follow his proposal. The major argument here is, he can just VETO to any >>>>> proposal to retain the codebase as the way he prefers to, which I don't >>>>> believe is a correct usage of VETO. >>>>> >>>>> If we just revert the change of removal of config, this is "really" >>>>> neutral neither my proposal nor his proposal. Do we really want to do so? >>>>> >>>>> On Mon, Mar 17, 2025 at 10:55 AM Holden Karau <holden.ka...@gmail.com> >>>>> wrote: >>>>> >>>>>> First let me start with my key hope: >>>>>> >>>>>> We find a way to compromise and have the veto withdrawn rather than >>>>>> overridden. >>>>>> >>>>>> From what I understand of the change in question: >>>>>> >>>>>> So my understanding, and I may be over simplifying here but there are >>>>>> (at least) three technical paths forward (migration guide, legacy config >>>>>> with vendor string in it, non-vendor specific string legacy config), a >>>>>> PMC >>>>>> member vetoed one of them (named vendor legacy config) because he >>>>>> thought a >>>>>> different approach was better (migration guide) as they were worried that >>>>>> carrying that legacy config forward would encourage bad coding standards >>>>>> (eg we would add more vendor named config flags). To me that seems like a >>>>>> valid concern. >>>>>> >>>>>> My reasoning: >>>>>> >>>>>> Thinking back at other VETOs that I’ve been involved with in this >>>>>> project (DSV2, graceful decom, etc) this seems to meet the same bar. Hell >>>>>> we’ve had plenty of vetos that didn’t offer an alternative. >>>>>> >>>>>> My personal understanding of where the bar for “ >>>>>> a technical justification showing why the change is bad” concern is >>>>>> pretty much “any not factually incorrect reasoning”, the text doesn’t >>>>>> have >>>>>> any particular “bar” for the level of “badness” and I think it’s up to >>>>>> the >>>>>> PMC member exercising the veto to use their judgement. >>>>>> >>>>>> In closing, I feel like the path we’re going down (overriding a veto) >>>>>> is not healthy for the project. >>>>>> >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>> Pronouns: she/her >>>>>> >>>>>> >>>>>> On Sun, Mar 16, 2025 at 6:28 PM Jungtaek Lim < >>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>> >>>>>>> Holden, I believe you should already know "both" approaches are >>>>>>> "technically" correct. It's not about which one you have a preference >>>>>>> for, >>>>>>> no, this VOTE is not intended to extend the debate. >>>>>>> >>>>>>> Again, what you are encouraged to do here is, not exposing your >>>>>>> preference of two approaches, but exposing your "technically valid" >>>>>>> concern >>>>>>> of my approach, backed by Dongjoon's veto (most likely you want to quote >>>>>>> Dongjoon's post). This is very simple and I'm not sure you are doing >>>>>>> exactly what the VOTE requires. >>>>>>> >>>>>>> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <holden.ka...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> -1 (binding) — to me it doesn’t matter that the cost is low if the >>>>>>>> objection is technical then I think we need to respect the veto. There >>>>>>>> is a >>>>>>>> fundamental disagreement as to what the correct technical way to >>>>>>>> address >>>>>>>> this problem is (removal + documentation vs legacy config) and a PMC >>>>>>>> member >>>>>>>> has vetoed the legacy config option. >>>>>>>> >>>>>>>> I think I disagree with Mark on the assertion that the veto needs >>>>>>>> to have “substantial technical concern,” but rather a valid concern. I >>>>>>>> think in addition to the veto they’ve also gone above and beyond >>>>>>>> providing >>>>>>>> alternative ways to accomplish this. >>>>>>>> >>>>>>>> On a personal level: >>>>>>>> >>>>>>>> I am optimistic we can unblock the release but I think it’s >>>>>>>> important to err on the side of respecting the veto here in the >>>>>>>> interest of >>>>>>>> perceived fairness *especially* because of vendor aspects. >>>>>>>> >>>>>>>> To be clear I’ve worked at most of these companies (and many of the >>>>>>>> people) and I’m not ascribing malice to anyone in this, I think >>>>>>>> mistakes >>>>>>>> happen (god knows I’ve had a fair share). I think we’re all doing our >>>>>>>> best >>>>>>>> here and would ask that we show everyone understanding regardless of >>>>>>>> the >>>>>>>> outcome. >>>>>>>> >>>>>>>> Sending hugs and good vibes to y’all. >>>>>>>> >>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>> Pronouns: she/her >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau < >>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Given it’s the weekend maybe let’s give folks at least one full >>>>>>>>> work day. >>>>>>>>> >>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>> Pronouns: she/her >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra < >>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Quick administrative note: I don't see any reason why this vote >>>>>>>>>> should >>>>>>>>>> take a long time, so I expect to close the process and tally the >>>>>>>>>> votes >>>>>>>>>> in not much more than 48 hours. >>>>>>>>>> >>>>>>>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra < >>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > There has been enough discussion on this topic already, so I >>>>>>>>>> think >>>>>>>>>> > that an immediate vote on the validity of Dongjoon's technical >>>>>>>>>> > justification for his veto of the "Retain migration logic ... >>>>>>>>>> in Spark >>>>>>>>>> > 4.0.x" proposal is in order. That technical justification has >>>>>>>>>> been >>>>>>>>>> > called into question, and the guidance at >>>>>>>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it >>>>>>>>>> to the >>>>>>>>>> > PMC to determine whether the technical justification is valid: >>>>>>>>>> "In >>>>>>>>>> > case of doubt, deciding whether a technical justification is >>>>>>>>>> valid is >>>>>>>>>> > up to the PMC." As such, only PMC votes will decide the outcome >>>>>>>>>> of >>>>>>>>>> > this vote. This is neither a vote on a code change itself not a >>>>>>>>>> vote >>>>>>>>>> > on whether a package is ready for release, so it a procedural >>>>>>>>>> vote on >>>>>>>>>> > whether the technical justification is valid. As such, the vote >>>>>>>>>> will >>>>>>>>>> > be decided by a simple majority where +1 votes hold that the >>>>>>>>>> technical >>>>>>>>>> > justification is not valid and -1 votes hold that the technical >>>>>>>>>> > justification is valid. >>>>>>>>>> > >>>>>>>>>> > I would request that at least PMC members post more than just a >>>>>>>>>> naked >>>>>>>>>> > vote, but instead endeavor to give some reason why they have >>>>>>>>>> assessed >>>>>>>>>> > the technical justification as they have. I'll start: >>>>>>>>>> > >>>>>>>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I >>>>>>>>>> must >>>>>>>>>> > confess to still not being entirely clear on what is his >>>>>>>>>> technical >>>>>>>>>> > justification for that veto. I see claims that including an >>>>>>>>>> admonition >>>>>>>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 >>>>>>>>>> is >>>>>>>>>> > required to maintain the integrity of already existing data >>>>>>>>>> streams, >>>>>>>>>> > and I see assertions about the maintenance burden that >>>>>>>>>> including the >>>>>>>>>> > migration logic would impose on future Spark versions, but I >>>>>>>>>> don't >>>>>>>>>> > think that I see any other technical objections. I do not >>>>>>>>>> believe that >>>>>>>>>> > the claimed technical justification is valid. >>>>>>>>>> > >>>>>>>>>> > In requiring that a veto of a code change be accompanied by a >>>>>>>>>> > technical justification for the veto, the Apache Voting Process >>>>>>>>>> states >>>>>>>>>> > that: "To prevent vetoes from being used capriciously, the >>>>>>>>>> voter must >>>>>>>>>> > provide with the veto a technical justification showing why the >>>>>>>>>> change >>>>>>>>>> > is bad (opens a security exposure, negatively affects >>>>>>>>>> performance, >>>>>>>>>> > etc. ). A veto without a justification is invalid and has no >>>>>>>>>> weight." >>>>>>>>>> > This strongly implies that there must be something objectively >>>>>>>>>> wrong >>>>>>>>>> > with the proposed code change in that it causes significant >>>>>>>>>> harm in >>>>>>>>>> > the way of opening a security exposure, negatively affecting >>>>>>>>>> > performance, or presumably other significant user harms or >>>>>>>>>> perhaps >>>>>>>>>> > even developer burdens. >>>>>>>>>> > >>>>>>>>>> > The proposed addition of the migration logic to Spark 4.0.x >>>>>>>>>> does not >>>>>>>>>> > cause any harm to Spark's users. For many users, those not using >>>>>>>>>> > streaming data, the change will have no effect. For streaming >>>>>>>>>> users >>>>>>>>>> > the change will be beneficial, not harmful. >>>>>>>>>> > >>>>>>>>>> > Neither do I find the claim of excessive, ongoing developer >>>>>>>>>> burden to >>>>>>>>>> > be persuasive. The changes are tiny and easily maintained -- in >>>>>>>>>> fact, >>>>>>>>>> > it wouldn't surprise me if no further changes to this migration >>>>>>>>>> logic >>>>>>>>>> > would be needed for a very long time. >>>>>>>>>> > >>>>>>>>>> > Some of what we are left with is just an expression of >>>>>>>>>> preference for >>>>>>>>>> > a technical alternative to the migration logic -- i.e. >>>>>>>>>> including in >>>>>>>>>> > the release notes an admonition to first upgrade to 3.5.5. But >>>>>>>>>> the >>>>>>>>>> > Apache Voting Process does not say that in the face of code >>>>>>>>>> > alternatives A and B, a qualified voter is justified in vetoing >>>>>>>>>> A if >>>>>>>>>> > they prefer B. Instead, the Voting Process strongly implies that >>>>>>>>>> > something more is needed to justify a veto, as I've already >>>>>>>>>> covered. >>>>>>>>>> > Thus I don't find Dongjoon's preference for the release notes >>>>>>>>>> option >>>>>>>>>> > to be adequate justification for the veto. >>>>>>>>>> > >>>>>>>>>> > The only remaining question I see is whether including >>>>>>>>>> "databricks" in >>>>>>>>>> > the Apache Code is ever allowed or if any such instance must be >>>>>>>>>> > expunged as soon as possible. I am not aware of any ASF policy >>>>>>>>>> that >>>>>>>>>> > strictly forbids the mention of a vendor in Apache code for any >>>>>>>>>> > reason, even if that vendor has a product based on Apache code, >>>>>>>>>> even >>>>>>>>>> > if that vendor enjoys a uniquely influential position vis a vis >>>>>>>>>> some >>>>>>>>>> > Apache code or project. Certainly the PMC has a duty to see to >>>>>>>>>> it that >>>>>>>>>> > neither Databricks nor any other vendor exercises influence or >>>>>>>>>> control >>>>>>>>>> > over Apache Spark outside of the established Apache process, >>>>>>>>>> but the >>>>>>>>>> > proposed migration code changes do not advantage Databricks -- >>>>>>>>>> if >>>>>>>>>> > anything they remove a minor avenue of influence, and simply >>>>>>>>>> need to >>>>>>>>>> > mention "databricks" once in order match and transform a >>>>>>>>>> configuration >>>>>>>>>> > into a vendor neutral equivalent. While not optimal, I can't >>>>>>>>>> find such >>>>>>>>>> > a one-time inclusion of "databricks" to be truly offensive to >>>>>>>>>> any >>>>>>>>>> > non-technical policy concern -- certainly not offensive to the >>>>>>>>>> point >>>>>>>>>> > that it outweighs the user advantage of including the migration >>>>>>>>>> logic >>>>>>>>>> > in Spark 4.0.x. >>>>>>>>>> > >>>>>>>>>> > In summary, I do not find Dongjoon's given technical >>>>>>>>>> justification to >>>>>>>>>> > be valid relative to the Apache requirements for a veto of a >>>>>>>>>> code >>>>>>>>>> > change, so I must vote... >>>>>>>>>> > >>>>>>>>>> > +1 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>> >>>>>>>>>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >
