Before I cast my vote here, I'd like to highlight one thing: As the release manager of Apache Spark 4.0.0, I was not notified about the breaking change of renaming an already-released configuration: https://github.com/apache/spark/pull/49897 . Note that the previous VOTE from Dongjoon was about Apache Spark 3.5.5, which means there is no consensus about what we should do for 4.0.0 yet. I think it's fair for me to ask to revert the breaking change and unblock 4.0.0, which is a common practice of how we handle breaking changes in Apache Spark and I don't think I need a VOTE for it.
Of course, none of us want to keep the misnamed configuration in 4.0.0, and it’s clear to me that applying the “configuration deprecation” approach from 3.5.5 to 4.0.0 is the best path forward. I don’t believe Dongjoon’s veto has valid technical justification, so I’m +1 on this vote. Thanks, Wenchen On Mon, Mar 17, 2025 at 7:27 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > I was trying hard to stay away from this VOTE, but I should have reminded > everyone about "what" we are going to VOTE. > > Dongjoon casted a VETO against code change VOTE. That VETO is described in > ASF Voting Process page: > > https://www.apache.org/foundation/voting.html#Veto > > A -1 vote by a qualified voter stops a code-modification proposal in its >> tracks. This constitutes a veto, and it cannot be overruled nor overridden >> by anyone. Vetoes stand until and unless the individual withdraws their >> veto. >> >> To prevent vetoes from being used capriciously, the voter must provide >> with the veto a technical justification showing why the change is bad >> (opens a security exposure, negatively affects performance, etc. ). A veto >> without a justification is invalid and has no weight. > > > For sure, the technical justification must be "objective", otherwise it > means if I'm a PMC member I can veto everything if I don't like it. > > The main argument here about "vendor name in the codebase" is NOT > something we have ever seen disallowing this in ASF policy. If there is > evidence, it will immediately kill the two VOTEs as it is enough objective > argument. But no one was able to bring this up. Please remember, the fact > "vendor name in the codebase is bad for any reason", is proven to be NOT an > "objective" claim, otherwise how the DISCUSS and VOTE were almost passing > with support from PMC members? > > I really suggest everyone who casts a vote in this VOTE thread, to be > based on "objective" rationale. For example, we tend to consider < 10 lines > of code to be very trivial to maintain, so the argument of maintenance > burden does not apply here. Like this. > > > On Sun, Mar 16, 2025 at 8:38 AM Mark Hamstra <markhams...@gmail.com> > wrote: > >> There has been enough discussion on this topic already, so I think >> that an immediate vote on the validity of Dongjoon's technical >> justification for his veto of the "Retain migration logic ... in Spark >> 4.0.x" proposal is in order. That technical justification has been >> called into question, and the guidance at >> https://www.apache.org/foundation/glossary.html#Veto leaves it to the >> PMC to determine whether the technical justification is valid: "In >> case of doubt, deciding whether a technical justification is valid is >> up to the PMC." As such, only PMC votes will decide the outcome of >> this vote. This is neither a vote on a code change itself not a vote >> on whether a package is ready for release, so it a procedural vote on >> whether the technical justification is valid. As such, the vote will >> be decided by a simple majority where +1 votes hold that the technical >> justification is not valid and -1 votes hold that the technical >> justification is valid. >> >> I would request that at least PMC members post more than just a naked >> vote, but instead endeavor to give some reason why they have assessed >> the technical justification as they have. I'll start: >> >> Despite all of the discussion related to Dongjoon's -1 vote, I must >> confess to still not being entirely clear on what is his technical >> justification for that veto. I see claims that including an admonition >> in the Spark 4.0.x release notes that a prior upgrade to 3.5.5 is >> required to maintain the integrity of already existing data streams, >> and I see assertions about the maintenance burden that including the >> migration logic would impose on future Spark versions, but I don't >> think that I see any other technical objections. I do not believe that >> the claimed technical justification is valid. >> >> In requiring that a veto of a code change be accompanied by a >> technical justification for the veto, the Apache Voting Process states >> that: "To prevent vetoes from being used capriciously, the voter must >> provide with the veto a technical justification showing why the change >> is bad (opens a security exposure, negatively affects performance, >> etc. ). A veto without a justification is invalid and has no weight." >> This strongly implies that there must be something objectively wrong >> with the proposed code change in that it causes significant harm in >> the way of opening a security exposure, negatively affecting >> performance, or presumably other significant user harms or perhaps >> even developer burdens. >> >> The proposed addition of the migration logic to Spark 4.0.x does not >> cause any harm to Spark's users. For many users, those not using >> streaming data, the change will have no effect. For streaming users >> the change will be beneficial, not harmful. >> >> Neither do I find the claim of excessive, ongoing developer burden to >> be persuasive. The changes are tiny and easily maintained -- in fact, >> it wouldn't surprise me if no further changes to this migration logic >> would be needed for a very long time. >> >> Some of what we are left with is just an expression of preference for >> a technical alternative to the migration logic -- i.e. including in >> the release notes an admonition to first upgrade to 3.5.5. But the >> Apache Voting Process does not say that in the face of code >> alternatives A and B, a qualified voter is justified in vetoing A if >> they prefer B. Instead, the Voting Process strongly implies that >> something more is needed to justify a veto, as I've already covered. >> Thus I don't find Dongjoon's preference for the release notes option >> to be adequate justification for the veto. >> >> The only remaining question I see is whether including "databricks" in >> the Apache Code is ever allowed or if any such instance must be >> expunged as soon as possible. I am not aware of any ASF policy that >> strictly forbids the mention of a vendor in Apache code for any >> reason, even if that vendor has a product based on Apache code, even >> if that vendor enjoys a uniquely influential position vis a vis some >> Apache code or project. Certainly the PMC has a duty to see to it that >> neither Databricks nor any other vendor exercises influence or control >> over Apache Spark outside of the established Apache process, but the >> proposed migration code changes do not advantage Databricks -- if >> anything they remove a minor avenue of influence, and simply need to >> mention "databricks" once in order match and transform a configuration >> into a vendor neutral equivalent. While not optimal, I can't find such >> a one-time inclusion of "databricks" to be truly offensive to any >> non-technical policy concern -- certainly not offensive to the point >> that it outweighs the user advantage of including the migration logic >> in Spark 4.0.x. >> >> In summary, I do not find Dongjoon's given technical justification to >> be valid relative to the Apache requirements for a veto of a code >> change, so I must vote... >> >> +1 >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
