I believe he intended to VETO this change for Spark 4, of course if Dongjoon did not (or no longer intends to) then this VOTE becomes moot. I think bringing up 3.5.5 confuses the issue -- this vote thread is very clearly about VETOing the change for Spark 4.
I think that accusing each-other of acting in bad faith is unproductive to resolving this dispute. On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Holden and Dongjoon, > > Let me make this vote super simple. I never got the answer from Dongjoon > about this question. This is super important because if he's casting veto > "to block", it is a strong indication that he was intended to play with me, > which I am seriously considering escalating the problem (If this is true, > it's no longer just a justification of vote, but someone's > behavioral issue). > > https://github.com/apache/spark/pull/49983 > > I might be missing another timeline, but, if you follow the conversation > here, there are some facts: > > 1. Dongjoon "knew" we were never decided about the direction of Spark > 4.0.0 behavior. (link > <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>) > 2. Dongjoon "agreed" my proposal is technically correct. (link > <https://github.com/apache/spark/pull/49983#issuecomment-2676531485>) > 3. Dongjoon "agreed" to hear from the community about discussing my > proposal. (link > <https://github.com/apache/spark/pull/49983#issuecomment-2696064545>) > > Worth clarifying, 3 happened after we discussed the removal of "config". > Dongjoon continuously mixed up the fact - while we were in agreement of > removal of config, removal of migration logic was definitely left to open > question. Let me give the VOTE Dongjoon drove and made it pass. > > https://lists.apache.org/thread/6nn76olr65b8zfgzdcbtr9f6o98451o5 > > This was totally about 3.5.5. If Dongjoon thinks this simply applies to > Spark 4.0.0+, it's not, no? > > Also, let's revisit the discussion we were discussing about removal of > config. > > https://lists.apache.org/thread/qxqzt7wbtdyxp17d7s1rxhnrswdccsgb > > Dongjoon clearly stated that we only make a consensus about Spark 3.5.5, > and we can continue discussion about the proper behavior in Spark 4.0.0. > That is the rationale I drove my own discussion. I can be corrected, but > there is NO discussion/vote w.r.t this topic AFAIK. > > Dongjoon, now it's your time to prove there is a valid reason to change > your mind during this time frame. If the above are all true, you are > already indicating that you can never cast a veto. (Or show me the evidence > of how you change your mind for which reason.) If any of the above are > something you intended to not tell the truth, I am really not sure your > comment will be truthful I can follow. Especially, if you did not tell the > truth from 3, e.g. you let me go and discuss while you were intended to > block me in any phase, this is a strong indication that you intend to play > with me and the community (or even ASF) has to know that. > > Do not evade the root question. > > On Mon, Mar 17, 2025 at 6:32 AM Holden Karau <holden.ka...@gmail.com> > wrote: > >> -1 (binding) — to me it doesn’t matter that the cost is low if the >> objection is technical then I think we need to respect the veto. There is a >> fundamental disagreement as to what the correct technical way to address >> this problem is (removal + documentation vs legacy config) and a PMC member >> has vetoed the legacy config option. >> >> I think I disagree with Mark on the assertion that the veto needs to have >> “substantial technical concern,” but rather a valid concern. I think in >> addition to the veto they’ve also gone above and beyond providing >> alternative ways to accomplish this. >> >> On a personal level: >> >> I am optimistic we can unblock the release but I think it’s important to >> err on the side of respecting the veto here in the interest of perceived >> fairness *especially* because of vendor aspects. >> >> To be clear I’ve worked at most of these companies (and many of the >> people) and I’m not ascribing malice to anyone in this, I think mistakes >> happen (god knows I’ve had a fair share). I think we’re all doing our best >> here and would ask that we show everyone understanding regardless of the >> outcome. >> >> Sending hugs and good vibes to y’all. >> >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >> >> On Sat, Mar 15, 2025 at 5:07 PM Holden Karau <holden.ka...@gmail.com> >> wrote: >> >>> Given it’s the weekend maybe let’s give folks at least one full work day. >>> >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>> >>> On Sat, Mar 15, 2025 at 4:44 PM Mark Hamstra <markhams...@gmail.com> >>> wrote: >>> >>>> Quick administrative note: I don't see any reason why this vote should >>>> take a long time, so I expect to close the process and tally the votes >>>> in not much more than 48 hours. >>>> >>>> On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra <markhams...@gmail.com> >>>> wrote: >>>> > >>>> > There has been enough discussion on this topic already, so I think >>>> > that an immediate vote on the validity of Dongjoon's technical >>>> > justification for his veto of the "Retain migration logic ... in Spark >>>> > 4.0.x" proposal is in order. That technical justification has been >>>> > called into question, and the guidance at >>>> > https://www.apache.org/foundation/glossary.html#Veto leaves it to the >>>> > PMC to determine whether the technical justification is valid: "In >>>> > case of doubt, deciding whether a technical justification is valid is >>>> > up to the PMC." As such, only PMC votes will decide the outcome of >>>> > this vote. This is neither a vote on a code change itself not a vote >>>> > on whether a package is ready for release, so it a procedural vote on >>>> > whether the technical justification is valid. As such, the vote will >>>> > be decided by a simple majority where +1 votes hold that the technical >>>> > justification is not valid and -1 votes hold that the technical >>>> > justification is valid. >>>> > >>>> > I would request that at least PMC members post more than just a naked >>>> > vote, but instead endeavor to give some reason why they have assessed >>>> > the technical justification as they have. I'll start: >>>> > >>>> > Despite all of the discussion related to Dongjoon's -1 vote, I must >>>> > confess to still not being entirely clear on what is his technical >>>> > justification for that veto. I see claims that including an admonition >>>> > in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is >>>> > required to maintain the integrity of already existing data streams, >>>> > and I see assertions about the maintenance burden that including the >>>> > migration logic would impose on future Spark versions, but I don't >>>> > think that I see any other technical objections. I do not believe that >>>> > the claimed technical justification is valid. >>>> > >>>> > In requiring that a veto of a code change be accompanied by a >>>> > technical justification for the veto, the Apache Voting Process states >>>> > that: "To prevent vetoes from being used capriciously, the voter must >>>> > provide with the veto a technical justification showing why the change >>>> > is bad (opens a security exposure, negatively affects performance, >>>> > etc. ). A veto without a justification is invalid and has no weight." >>>> > This strongly implies that there must be something objectively wrong >>>> > with the proposed code change in that it causes significant harm in >>>> > the way of opening a security exposure, negatively affecting >>>> > performance, or presumably other significant user harms or perhaps >>>> > even developer burdens. >>>> > >>>> > The proposed addition of the migration logic to Spark 4.0.x does not >>>> > cause any harm to Spark's users. For many users, those not using >>>> > streaming data, the change will have no effect. For streaming users >>>> > the change will be beneficial, not harmful. >>>> > >>>> > Neither do I find the claim of excessive, ongoing developer burden to >>>> > be persuasive. The changes are tiny and easily maintained -- in fact, >>>> > it wouldn't surprise me if no further changes to this migration logic >>>> > would be needed for a very long time. >>>> > >>>> > Some of what we are left with is just an expression of preference for >>>> > a technical alternative to the migration logic -- i.e. including in >>>> > the release notes an admonition to first upgrade to 3.5.5. But the >>>> > Apache Voting Process does not say that in the face of code >>>> > alternatives A and B, a qualified voter is justified in vetoing A if >>>> > they prefer B. Instead, the Voting Process strongly implies that >>>> > something more is needed to justify a veto, as I've already covered. >>>> > Thus I don't find Dongjoon's preference for the release notes option >>>> > to be adequate justification for the veto. >>>> > >>>> > The only remaining question I see is whether including "databricks" in >>>> > the Apache Code is ever allowed or if any such instance must be >>>> > expunged as soon as possible. I am not aware of any ASF policy that >>>> > strictly forbids the mention of a vendor in Apache code for any >>>> > reason, even if that vendor has a product based on Apache code, even >>>> > if that vendor enjoys a uniquely influential position vis a vis some >>>> > Apache code or project. Certainly the PMC has a duty to see to it that >>>> > neither Databricks nor any other vendor exercises influence or control >>>> > over Apache Spark outside of the established Apache process, but the >>>> > proposed migration code changes do not advantage Databricks -- if >>>> > anything they remove a minor avenue of influence, and simply need to >>>> > mention "databricks" once in order match and transform a configuration >>>> > into a vendor neutral equivalent. While not optimal, I can't find such >>>> > a one-time inclusion of "databricks" to be truly offensive to any >>>> > non-technical policy concern -- certainly not offensive to the point >>>> > that it outweighs the user advantage of including the migration logic >>>> > in Spark 4.0.x. >>>> > >>>> > In summary, I do not find Dongjoon's given technical justification to >>>> > be valid relative to the Apache requirements for a veto of a code >>>> > change, so I must vote... >>>> > >>>> > +1 >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
