It's OK, I just wanted to let him give the link whenever he makes the argument. It is NOT important that he brings up a new reasoning at this point. It must have been there in VOTE.
Also, I think we all agree that this is NOT a place of debate about having vendor names in the codebase. I think I gave plenty of time from DISCUSSION and VOTE to let PMC members chime in about their concerns. Let me remind - this VOTE is, to judge, of Dongjoon's VETO. Again, Mark clarified the criteria of VETO from quoting ASF page: > > In requiring that a veto of a code change be accompanied by a > technical justification for the veto, the Apache Voting Process states > that: "To prevent vetoes from being used capriciously, the voter must > provide with the veto a technical justification showing why the change > is bad (opens a security exposure, negatively affects performance, > etc. ). A veto without a justification is invalid and has no weight." > This strongly implies that there must be something objectively wrong > with the proposed code change in that it causes significant harm in > the way of opening a security exposure, negatively affecting > performance, or presumably other significant user harms or perhaps > even developer burdens. Anyone speaking for the judge, the above must be the central reasoning to do so. If the claim is sidetracked from this criteria of VETO, this just expands the previous VOTE which we intend to initiate this VOTE to "avoid" it. On Mon, Mar 17, 2025 at 7:40 AM Holden Karau <holden.ka...@gmail.com> wrote: > I believe he intended to VETO this change for Spark 4, of course if > Dongjoon did not (or no longer intends to) then this VOTE becomes moot. I > think bringing up 3.5.5 confuses the issue -- this vote thread is very > clearly about VETOing the change for Spark 4. > > I think that accusing each-other of acting in bad faith is unproductive to > resolving this dispute. > > On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Holden and Dongjoon, >> >> Let me make this vote super simple. I never got the answer from Dongjoon >> about this question. This is super important because if he's casting veto >> "to block", it is a strong indication that he was intended to play with me, >> which I am seriously considering escalating the problem (If this is true, >> it's no longer just a justification of vote, but someone's >> behavioral issue). >> >> https://github.com/apache/spark/pull/49983 >> >> I might be missing another timeline, but, if you follow the conversation >> here, there are some facts: >> >> 1. Dongjoon "knew" we were never decided about the direction of Spark >> 4.0.0 behavior. (link >> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>) >> 2. Dongjoon "agreed" my proposal is technically correct. (link >> <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>) >> 3. Dongjoon "agreed" to hear from the community about discussing my >> proposal. (link >> <https://github.com/apache/spark/pull/49983#issuecomment-2696064545>) >> >> Worth clarifying, 3 happened after we discussed the removal of "config". >> Dongjoon continuously mixed up the fact - while we were in agreement of >> removal of config, removal of migration logic was definitely left to open >> question. Let me give the VOTE Dongjoon drove and made it pass. >> >> https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5 >> >> This was totally about 3.5.5. If Dongjoon thinks this simply applies to >> Spark 4.0.0+, it's not, no? >> >> Also, let's revisit the discussion we were discussing about removal of >> config. >> >> https://lists.apache.org/thread/qxqzt7wbtdyxp17d7s1rxhnrswdccsgb >> >> Dongjoon clearly stated that we only make a consensus about Spark 3.5.5, >> and we can continue discussion about the proper behavior in Spark 4.0.0. >> That is the rationale I drove my own discussion. I can be corrected, but >> there is NO discussion/vote w.r.t this topic AFAIK. >> >> Dongjoon, now it's your time to prove there is a valid reason to change >> your mind during this time frame. If the above are all true, you are >> already indicating that you can never cast a veto. (Or show me the evidence >> of how you change your mind for which reason.) If any of the above are >> something you intended to not tell the truth, I am really not sure your >> comment will be truthful I can follow. Especially, if you did not tell the >> truth from 3, e.g. you let me go and discuss while you were intended to >> block me in any phase, this is a strong indication that you intend to play >> with me and the community (or even ASF) has to know that. >> >> Do not evade the root question. >> >> On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <holden.ka...@gmail.com> >> wrote: >> >>> -1 (binding) — to me it doesn’t matter that the cost is low if the >>> objection is technical then I think we need to respect the veto. There is a >>> fundamental disagreement as to what the correct technical way to address >>> this problem is (removal + documentation vs legacy config) and a PMC member >>> has vetoed the legacy config option. >>> >>> I think I disagree with Mark on the assertion that the veto needs to >>> have “substantial technical concern,” but rather a valid concern. I think >>> in addition to the veto they’ve also gone above and beyond providing >>> alternative ways to accomplish this. >>> >>> On a personal level: >>> >>> I am optimistic we can unblock the release but I think it’s important to >>> err on the side of respecting the veto here in the interest of perceived >>> fairness *especially* because of vendor aspects. >>> >>> To be clear I’ve worked at most of these companies (and many of the >>> people) and I’m not ascribing malice to anyone in this, I think mistakes >>> happen (god knows I’ve had a fair share). I think we’re all doing our best >>> here and would ask that we show everyone understanding regardless of the >>> outcome. >>> >>> Sending hugs and good vibes to y’all. >>> >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>> >>> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <holden.ka...@gmail.com> >>> wrote: >>> >>>> Given it’s the weekend maybe let’s give folks at least one full work >>>> day. >>>> >>>> Twitter: https://twitter.com/holdenkarau >>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> Pronouns: she/her >>>> >>>> >>>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <markhams...@gmail.com> >>>> wrote: >>>> >>>>> Quick administrative note: I don't see any reason why this vote should >>>>> take a long time, so I expect to close the process and tally the votes >>>>> in not much more than 48 hours. >>>>> >>>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <markhams...@gmail.com> >>>>> wrote: >>>>> > >>>>> > There has been enough discussion on this topic already, so I think >>>>> > that an immediate vote on the validity of Dongjoon's technical >>>>> > justification for his veto of the "Retain migration logic ... in >>>>> Spark >>>>> > 4.0.x" proposal is in order. That technical justification has been >>>>> > called into question, and the guidance at >>>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to >>>>> the >>>>> > PMC to determine whether the technical justification is valid: "In >>>>> > case of doubt, deciding whether a technical justification is valid is >>>>> > up to the PMC." As such, only PMC votes will decide the outcome of >>>>> > this vote. This is neither a vote on a code change itself not a vote >>>>> > on whether a package is ready for release, so it a procedural vote on >>>>> > whether the technical justification is valid. As such, the vote will >>>>> > be decided by a simple majority where +1 votes hold that the >>>>> technical >>>>> > justification is not valid and -1 votes hold that the technical >>>>> > justification is valid. >>>>> > >>>>> > I would request that at least PMC members post more than just a naked >>>>> > vote, but instead endeavor to give some reason why they have assessed >>>>> > the technical justification as they have. I'll start: >>>>> > >>>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must >>>>> > confess to still not being entirely clear on what is his technical >>>>> > justification for that veto. I see claims that including an >>>>> admonition >>>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is >>>>> > required to maintain the integrity of already existing data streams, >>>>> > and I see assertions about the maintenance burden that including the >>>>> > migration logic would impose on future Spark versions, but I don't >>>>> > think that I see any other technical objections. I do not believe >>>>> that >>>>> > the claimed technical justification is valid. >>>>> > >>>>> > In requiring that a veto of a code change be accompanied by a >>>>> > technical justification for the veto, the Apache Voting Process >>>>> states >>>>> > that: "To prevent vetoes from being used capriciously, the voter must >>>>> > provide with the veto a technical justification showing why the >>>>> change >>>>> > is bad (opens a security exposure, negatively affects performance, >>>>> > etc. ). A veto without a justification is invalid and has no weight." >>>>> > This strongly implies that there must be something objectively wrong >>>>> > with the proposed code change in that it causes significant harm in >>>>> > the way of opening a security exposure, negatively affecting >>>>> > performance, or presumably other significant user harms or perhaps >>>>> > even developer burdens. >>>>> > >>>>> > The proposed addition of the migration logic to Spark 4.0.x does not >>>>> > cause any harm to Spark's users. For many users, those not using >>>>> > streaming data, the change will have no effect. For streaming users >>>>> > the change will be beneficial, not harmful. >>>>> > >>>>> > Neither do I find the claim of excessive, ongoing developer burden to >>>>> > be persuasive. The changes are tiny and easily maintained -- in fact, >>>>> > it wouldn't surprise me if no further changes to this migration logic >>>>> > would be needed for a very long time. >>>>> > >>>>> > Some of what we are left with is just an expression of preference for >>>>> > a technical alternative to the migration logic -- i.e. including in >>>>> > the release notes an admonition to first upgrade to 3.5.5. But the >>>>> > Apache Voting Process does not say that in the face of code >>>>> > alternatives A and B, a qualified voter is justified in vetoing A if >>>>> > they prefer B. Instead, the Voting Process strongly implies that >>>>> > something more is needed to justify a veto, as I've already covered. >>>>> > Thus I don't find Dongjoon's preference for the release notes option >>>>> > to be adequate justification for the veto. >>>>> > >>>>> > The only remaining question I see is whether including "databricks" >>>>> in >>>>> > the Apache Code is ever allowed or if any such instance must be >>>>> > expunged as soon as possible. I am not aware of any ASF policy that >>>>> > strictly forbids the mention of a vendor in Apache code for any >>>>> > reason, even if that vendor has a product based on Apache code, even >>>>> > if that vendor enjoys a uniquely influential position vis a vis some >>>>> > Apache code or project. Certainly the PMC has a duty to see to it >>>>> that >>>>> > neither Databricks nor any other vendor exercises influence or >>>>> control >>>>> > over Apache Spark outside of the established Apache process, but the >>>>> > proposed migration code changes do not advantage Databricks -- if >>>>> > anything they remove a minor avenue of influence, and simply need to >>>>> > mention "databricks" once in order match and transform a >>>>> configuration >>>>> > into a vendor neutral equivalent. While not optimal, I can't find >>>>> such >>>>> > a one-time inclusion of "databricks" to be truly offensive to any >>>>> > non-technical policy concern -- certainly not offensive to the point >>>>> > that it outweighs the user advantage of including the migration logic >>>>> > in Spark 4.0.x. >>>>> > >>>>> > In summary, I do not find Dongjoon's given technical justification to >>>>> > be valid relative to the Apache requirements for a veto of a code >>>>> > change, so I must vote... >>>>> > >>>>> > +1 >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> >>>>> > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her >
