Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

FWIW, I am +1 on the proposal (though I missed the vote on this !) Regards, Mridul On Fri, Mar 14, 2025 at 1:31 AM Mridul Muralidharan wrote: > > I agree with Mark, imo this is a qualified veto. > We should give Dongjoon the opportunity to give his clarification, if any. > > I do realize this

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I agree with Mark, imo this is a qualified veto. We should give Dongjoon the opportunity to give his clarification, if any. I do realize this delays the RC process, but this deserves to be looked into carefully. Thanks, Mridul On Thu, Mar 13, 2025 at 9:35 PM Mark Hamstra wrote: > Absolutely

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

It is definitely his responsibility to explain why he casted -1. He explicitly casted his -1 in DISCUSS and I don't believe I hear anything. The only thing I heard is "users can upgrade Spark 3.5.5 before upgrading to Spark 4.0.0". How do you justify that this is a technical objection while we have

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

It is not Dongjoon's responsibility to clarify ASF policy for you. If you have ASF policy questions, there are ways to address them through the PMC, legal and the board, not by making demands on Dongjoon. I don't presume to speak for the whole PMC as to whether or not having "spark.databricks" appe

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Again, we have not spent 3 weeks on the matter at hand: whether Dongjoon's veto is valid. Please stop asserting irrelevant timeframes and extraneous issues. The end of this week appears more than adequate and fair to me. On Thu, Mar 13, 2025 at 9:46 PM Jungtaek Lim wrote: > > I love to hear what

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I love to hear what is the reasonable time here. If you say 1 week, it doesn't make sense at all. So what time do you suggest on the deadline? Will you be fine by the end of this week? Don't leave the status to be ambiguous. We already spent 3 weeks there. I don't want to let this be dragged. On

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

The relevant time window is since Dongjoon's veto was challenged, not any other that you choose to assert. It has been less than a day since that challenge. Dongjoon presented a prima facie correct veto to the proposal. The technical justification he gave was challenged or asserted to be invalid.

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

There is not clear support for the position in this vote. There is a clear effort to veto by Dongjoon. He provided some technical justification with his -1 vote. That justification has been called into question. We have not yet seen his response to that, and it has been less than 24 hours. Frankly

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

But can you please explain how we can be placed to be a fair situation? Let's say, assume we have the migration code in the current codebase. Dongjoon will never be able to remove the code, no? There are only two choices and I believe the codebase as it is is "accidentally" following his proposal.

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I am open to waiting for a day, but please be sure to remember that 3 weeks have passed and he had plenty of time to persuade people like I did. Also, I'd like to remind you that I did not attempt "just one time" to get his voice (yeah, persuade, actually). This is the post I sent to ask for revi

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Characterizing Dongjoon's position as just "agree to disagree" without any valid technical issue is your position. I have not seen any endorsement from him on list that this is a correct characterization of his position. I see recent questioning of whether Dongjoon's veto is justified by a valid t

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Actually, this has been initially triggered from 3 weeks ago, not just a week we have spent. https://github.com/apache/spark/pull/49983#issuecomment-2676531485 Mark, do you still want me to persuade Dongjoon while I clearly saw his stance on this on the VOTE thread? He can correct me, but from wha

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

As the release manager for 4.0, I’m awaiting a conclusion on this topic before cutting the next RC. There have been many debates, and I’m trying to understand what has happened so far. 1. A mistake was made, leading to a vendor name being included in the configuration released in Spark 3.5.4. 2. Do

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

This has been ongoing for a week, the vote has been open for 3 days, Dongjoon has replied today (not sure if you saw it), and I think this is all around in circles; I don't see any basis for waiting 24 hours (? where is this from?) I don't know if this is a code change vote - there is no code chang

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Absolutely not! This is clearly a vote on a code change, not on a procedural issue or a package release. The code change has been vetoed by a -1 vote by a qualified voter. On Thu, Mar 13, 2025 at 6:58 PM Jungtaek Lim wrote: > > Likewise I said, I'm concluding the VOTE since we ensure the criteri

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

This vote has not passed. The proposed code change has been vetoed by a qualified voter. The validity of that veto has been called into question since "the voter must provide with the veto a technical justification showing why the change is bad (opens a security exposure, negatively affects perfor

Unsubscribe

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Likewise I said, I'm concluding the VOTE since we ensure the criteria (3 +1 binding, 1 -1 binding, and also +1s from non-binding). I don't consider -1 as a veto as I explained, as we should have multiple -1s if we go for VOTE with the current codebase. (+1 in this proposal is effectively -1 in ano

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Will this nanosecond timestamp be a fixed-size (10 bytes) binary in UnsafeRow and ColumnVector? On Thu, Mar 13, 2025 at 4:57 PM Jungtaek Lim wrote: > Hi, would you mind allowing comments on the doc? Thanks! > > On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > >> Hello everybody, >> >> I would li

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hi, would you mind allowing comments on the doc? Thanks! On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > Hello everybody, > > I would like to start a discussion on SPARK-50532 > to enable Spark to > support nanoseconds. Here attached the spip d

[Discuss] SPIP: Support NanoSecond Timestamps

Hello everybody, I would like to start a discussion on SPARK-50532 to enable Spark to support nanoseconds. Here attached the spip doc . Huaxin was

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I understand the situation of "agree to disagree", but I don't believe veto can be used like that. It's just -1 from the minority. Here are questions I want to get answered. Where is the evidence that having a vendor name in the codebase is violating ASF policy? Again, I see "Apple" to be used as

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Also, I don't believe considering -1 as veto makes sense here, because his proposal is "somehow" (I'd rather say "accidentally") in the current codebase and we hadn't had any discussion with that proposal. So if we kill the VOTE and do nothing, it's effectively saying +1 to his proposal, which make

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I do believe there are two ways of considering -1 vote. Valid -1 votes are not restricted to technical objections, but in that case, it must not be considered as veto, otherwise we will end up disturbing ourselves. It is just an ideal world where we can make consensus on any topic, no, it can't be.

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Valid -1 votes are not restricted to technical objections. On Thu, Mar 13, 2025 at 7:28 AM Sean Owen wrote: > > I'm not sure if a VOTE is appropriate here, but I also do not see any valid > technical objection here. I don't think this can be considered a valid 'veto' > even if we were thinking

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I'm not sure if a VOTE is appropriate here, but I also do not see any valid technical objection here. I don't think this can be considered a valid 'veto' even if we were thinking of it that way. I think there are other non-technical factors influencing this position. I believe we proceed with Jungt

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

We are having this vote to give clarity by keeping all records of the community decisions and stances during building a community consensus. All votes are important and counted. To Jungtaek, I already casted my veto properly and have been tracking the thread. You don't need to say to me to revisit

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Thanks to everyone who participated and voted! Now I can technically conclude the VOTE, but I'm willing to wait till US daytime tomorrow, to give some time for Dongjoon to revisit this. I'll conclude the vote around 6PM PST tomorrow regardless of his vote. It's ideal to see us have no -1, but hav

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

forgot to mention in my last reply, my stance is +1 Jie Yang On 2025/03/13 07:08:12 Russell Jurney wrote: > Sure, +1 non-binding. > > On Wed, Mar 12, 2025 at 11:18 PM Jungtaek Lim > wrote: > > > Russell, > > > > Of course, we hear people' voices who aren't having binding votes as well. > > Per

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Sure, +1 non-binding. On Wed, Mar 12, 2025 at 11:18 PM Jungtaek Lim wrote: > Russell, > > Of course, we hear people' voices who aren't having binding votes as well. > Personally I think it's more important than committers/PMC members' VOTE > this time since we can be biased and be far from user

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Dongjoon, I wonder whether I can influence you to revisit what has happened. We are here because we have two different approaches where we have agreed to disagree on the approach. We posted the discussion in dev@ because we want to hear from 3rd eyes. In DISCUSSION and VOTE threads, I'm seeing sup