Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Just one hope, I believe I have said I will hear about support of migration logic for next release. The scope of the VOTE is nothing beyond 4.0.x. Please do not interpret this in your own way. I continuously see that I am attacked by what I am not saying and I have to prove that I haven't said. Ple

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Dongjoon, I'm now OK with whatever you think, but I argue your vote is technically moot since it's about your vote justification, and I have no binding vote to counter you. Let's be fair. On Sun, Mar 16, 2025 at 3:07 PM Dongjoon Hyun wrote: > Thank you for focusing on this, Mark. > > I also agr

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Thank you for focusing on this, Mark. I also agree with you that this should be decided by the Apache Spark PMC and appreciate the effort to help us move forward in the Apache way. As you mentioned, there is no ASF policy. That's true. > I am not aware of any ASF policy that strictly forbids th

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

> Isn't it a bit excessive to talk about "making the voice of the community very clear" when only a very small fraction of Spark users have participated in the discussion or cast their votes? This has been an ongoing issue for every single project. You get less votes if you initiate VOTE for more

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Isn't it a bit excessive to talk about "making the voice of the community very clear" when only a very small fraction of Spark users have participated in the discussion or cast their votes? That said, I read the initial emails and understood the proposal. Since it seemed obvious and straightforwar

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Guys, please take a second look at the proposal before chiming in. The config will be removed in 4.0 regardless of this VOTE. It is really really just a string. 2025년 3월 16일 (일) 오전 6:58, Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com>님이 작성: > I agree with Holden regarding Spark 4 being the

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Ok guys great discussion all around. it would be* interesting* to observe sentiments by participants in this thread. The source is from spark dev archive for this topic as attached. The code uses Python with the relevant libraries see the code and imbedded explanation *Approach for Sentiment Ana

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Slight correction (to avoid nitpick): “vote” in the last sentence is “veto”. I think community will understand correctly, but just to avoid any voice making this to be sidetracked. 2025년 3월 16일 (일) 오전 11:20, Jungtaek Lim 님이 작성: > For sure, I’m +1 (non-binding). > > I believe I don’t need to expla

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Thanks Mark for trying to make the voice from community very clear and be recorded. Now community is required to just validate Dongjoon’s veto based on his last post and there is no longer a concern about vote process with your proposal. If we agree that Dongjoon’s justification is moot, I’ll need

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

For sure, I’m +1 (non-binding). I believe I don’t need to explain more and I spent whole weekend and we have a history about my justification in the history of mailing list. I’m open to summary my justification again, but as a tl;dr, I have a strong evidence that he knew we never had a consensus

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1Sent from my iPhoneOn Mar 14, 2025, at 11:28 PM, Szehon Ho wrote:+1 to the idea as well, as Iceberg V3 is coming with time with nanos, and Spark would not be able to read this type without this.ThanksSzehonOn Fri, Mar 14, 2025 at 3:34 PM Wenchen Fan wrote:In general, I thi

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Given it’s the weekend maybe let’s give folks at least one full work day. Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Quick administrative note: I don't see any reason why this vote should take a long time, so I expect to close the process and tally the votes in not much more than 48 hours. On Sat, Mar 15, 2025 at 4:35 PM Mark Hamstra wrote: > > There has been enough discussion on this topic already, so I think

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I don’t think we quite need to involve the board yet. There is still the open question of whether the technical justification for the veto is valid, and it is purely up to the PMC to resolve those doubts. I don’t understand the apparent reluctance to call for resolution of that question even thoug

[VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

There has been enough discussion on this topic already, so I think that an immediate vote on the validity of Dongjoon's technical justification for his veto of the "Retain migration logic ... in Spark 4.0.x" proposal is in order. That technical justification has been called into question, and the g

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I agree with Holden regarding Spark 4 being the perfect time to drop this configuration. El sáb, 15 mar 2025, 22:33, Holden Karau escribió: > My $0.02: > > I do not believe that this vote has passed. I believe there is a valid > veto. On a personal level from a migration point of view I think Sp

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

My $0.02: I do not believe that this vote has passed. I believe there is a valid veto. On a personal level from a migration point of view I think Spark 4 is the perfect time to drop this configuration. Given the disagreement of if this is a valid veto I think we should pause until the board has b

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

This is my gist Mark from your passionate language I gather you see this as a "Code Change" veto. Your reasoning seems to be straightforward, i.e. the vote's purpose is to decide whether to add code (migration logic) to the Spark 4.0 branch. In your view, the outcome of the vote directly alters th

Re: Re: [Discuss] SPIP: Support NanoSecond Timestamps

In general, I think it's good for Spark to support the common data types in the ecosystem, as it's the only way to fully integrate with the ecosystem. So +1. On Fri, Mar 14, 2025 at 8:56 AM 谭琦 wrote: > Updated. Thanks. > > On 2025/03/13 23:56:20 Jungtaek Lim wrote: > > Hi, would you mind allowin

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

That is utter nonsense, Sean! You do not have any authority to declare the matter concluded, and I will escalate to the board if you persist in this approach. The proposed code change has been vetoed. As I delineated previously, there are two and only two ways forward under the ASF Voting Process.

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Mark et al - this thread has gone on way too long. Everyone has expressed their opinion. The result stands. Anyone who is really upset about it, please escalate to the board or something, but, this thread and decision point has now concluded. On Sat, Mar 15, 2025 at 1:16 PM Mark Hamstra wrote:

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

You do not have the authority to declare Dongjoon's technical justification invalid. That is up to the PMC: "In case of doubt, deciding whether a technical justification is valid is up to the PMC." On Sat, Mar 15, 2025 at 6:20 AM Jungtaek Lim wrote: > > To summarize, the main arguments of both pr

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Once again, a vote procedure is not a procedural vote. This is unarguably a code change vote: the whole point of the vote is to free you to change the 4.0 branch to include migration logic code present in 3.5. We are not voting on whether a package is ready to release, nor are we voting on a proce

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

small missing on link: 4. I claimed I wanted to proceed with migration logic for branch-4.0 PR, and hadn't got any feedback except being told to wait for Spark 3.5.5 (link ). If you weren't open to my proposal, you should hav

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

UPDATE: We were having a discussion about the type of VOTE, since Dongjoon's -1 should be considered as a veto if we see this as a code change VOTE. Dongjoon clarified that he does not see this VOTE as a code change, hence he gave -1 but not intended to block the VOTE. That said, we have confirme

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Starting from my +1 (non-binding). In addition, I propose to retain migration logic till Spark 4.1.x and remove it in Spark 4.2.0. On Mon, Mar 10, 2025 at 9:44 PM Jungtaek Lim wrote: > Hi dev, > > Please vote to retain migration logic of incorrect `spark.databricks.*` > configuration in Spark 4

RE: Re: [Discuss] SPIP: Support NanoSecond Timestamps

Updated. Thanks. On 2025/03/13 23:56:20 Jungtaek Lim wrote: > Hi, would you mind allowing comments on the doc? Thanks! > > On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > > > Hello everybody, > > > > I would like to start a discussion on SPARK-50532 > >

Unsubscribe

Unsubscribe

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Apache Spark PMC always strongly recommends all 3.5 users to upgrade to the latest stable release via the official website. The main question seems quite different from the Apache Spark website. May I ask what is not safe to guide Spark 3.5.4 users to 3.5.5, Jungtaek? > The main question was, "whe

[VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

The vote passes with 7 +1s (3 binding +1s) and 1 -1s (1 binding -1s). Thanks to all who helped with the vote! I'm going to make a code change in branch-4.0 quickly so that we don't have to trigger another RC for Spark 4.0.0 just because of this. (* = binding) +1: - Sean R. Owen * - Jungtaek Lim -

Unsubscribe

Sent from my iPhone - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Unsubscribe

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Dongjoon, it is your responsibility to clarify your vote position since the vote is stalled as some people still claim your vote is veto. If you are really agreeing that I gained the consensus in the proper way, and your vote is really just for historical record, let's not waste more time by explic

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

> according to the ASF process, the Apache Spark community made the conclusion to unblock the Apache Spark 4.0.0 release with the AS-IS code with the improved Spark 4.0 migration guide because I provided a technical justification for my vote via the concrete alternative based on the existing Spark

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

To summarize, the main arguments of both proposals are "whether we can force users to upgrade to Spark 3.5.5 first before upgrading Spark 4.0.0" vs "we should include migration logic to Spark 4.0.0 because that is not realistic". Where is the "technical objection" here? If you say there was politic

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

> That's the reason why you proposed the vote procedure and we agreed. Didn’t you see the part “we agreed”? Who is we in the context? I don’t think he answered my questions - he explained his reasoning of his proposal which majorly does not agree with. You even said uou are not persuaded and I wa

Unsubscribe

Unsubscribe

Unsubscribe

Unsubscribe - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Once again, I have to object. Dongjoon said that the vote is a time limited procedure, not that the vote itself is a procedural vote as distinct from a code change vote or a package release vote. Frankly, this feels like you are trying to manipulate the vote procedure by misrepresenting Dongjoon,

Unsubscribe

Unsubscribe

Code formatting tech debt

Hi Spark devs, There seems to be a lot of code formatting tech debt. When I run "./dev/scalafmt" on the master branch it makes formatting changes on thousands of files. Is that expected or am I doing something wrong? If these files should be formatted we could add a formatting check to the PR to p

Unsubscribe

unsubscribe

Unsubscribe

unsubscribe

Unsubscribe

Unsubscribe

Re: Code formatting tech debt

I am not 100% sure, but I think you should run : ./dev/lint-scala . Some time back I also ran the target which you said, and resulted in plethora of files being modified. Then at that time realized, that target was run only for some specific modules ( I think connect..). Regards Asif On Fri